Platform on lo0 — Tech Blog

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

Sun, 31 May 2026 08:00:00 +0200

TL;DR

The two previous posts in this architectural trilogy settled the pieces: the seven layers of the LLM inference stack describe the components above the cluster, and the five maturity levels of the platform describe the strata below. This post settles the when: in what order each thing gets deployed when you start from zero, hardware bought, racks installed, physical cabling done, and you want to reach a cluster serving the first production token to a customer. Seven nominal phases F0 to F6 with no calendar commitments, organised by technical dependencies (you do not enter F3 without the F2 gate) and with an identifiable critical path. F0 hardware inventory and electrical/network connectivity. F1 bare metal OS + drivers + container runtime. F2 Kubernetes cluster with CNI and Ceph storage operational. F3 GitOps and infrastructure observability. F4 identity, TLS, secrets and policies. F5 GPU platform with LLM-aware observability. F6 the LLM stack operational and open to production traffic. For each phase: what gets built, what has to be ready first (dependencies between phases), the gate that validates its closure, and the typical trap that delays the critical path. The thesis: an on-premise LLM platform sinks far more often from sequencing badly than from choosing badly. The tools have all been invented; the order is the only thing every team rediscovers.

You are here: the seven phases and their dependencies

The phases do not run in pure series. F2 and F3 can be started at the same time to speed things up (installing Kubernetes and preparing the GitOps repo in parallel). F4 can overlap with the tail end of F3. F5 waits for F4 to close because GPU pods demand NetworkPolicy and RBAC from day one. F6 is a single atomic step: the cluster goes into production or it does not.

The red arrows are the critical path: the sequential bottleneck that no parallelism can shorten. The grey arrows are dependencies that admit partial overlap. Recognising where to overlap and where not is the difference between a three-month deployment and a six-month one for the same perimeter.

The analogy: the expedition to an eight-thousand-metre summit

An expedition to a high alpine summit is not a long trek. It is a series of camps that are set up in order, each with its altitude, its function and its validation gate: if you do not acclimatise properly at base camp, you cannot go up to C1 without risk; if C2 does not have its kitchen and its radio running, you cannot send people higher; if the summit push is attempted without porters in the high camps, there is no safe descent.

Greenfield deployment of an LLM platform works identically. F0 is the arrival of the material at base camp: boxes, sponsors, permits, first review. F1 is setting up an operational base camp: kitchen, tents, generator. F2 is the climb to C1: there is real altitude now (a k8s cluster running) and you breathe differently. F3 is C2: it adds communications, planning and operational acclimatisation. F4 is C3, the last night before the push: team roped up, oxygen ready, every protocol verified. F5 is the day of the summit push, intense effort, thin margins. F6 is the summit and the start of a safe descent: from here on the expedition is in day-to-day operation, no longer in construction.

The analogy holds two useful lessons: you do not skip a camp (going straight from base camp to the summit kills the team), and the gates are technical, not emotional (if the barometer says storm, you do not set out, however much enthusiasm there is). The platform team that follows those two rules reaches the summit. The one that negotiates them does not.

F0 — Hardware in the room: base camp

What gets built in this phase. An inventory of the hardware received (servers, switches, PDUs, BMC), racks assembled, electrical and data cabling terminated, physical labelling of each machine (rack/U/function), connectivity to the corporate network, management IPs assigned, BMC reachable over VPN with MFA, first ping to each node from the bastion.

Dependencies. None technical, this is the phase before the software. There are procurement dependencies (servers bought, switches bought), civil works dependencies (a room with enough cooling, a raised floor) and administrative ones (datacenter access for the technicians).

Validation gate that closes F0.

Every node appears in the inventory with (hostname, MAC, management IP, data IP, rack, U, function, owner).
Each node’s BMC responds to ipmitool power status and to the HTTPS UI from the management VPN.
The top-of-rack switch has its configuration versioned in git (even if there is no cluster GitOps yet, the switch configs are).
A command for h in $(cat hosts); do ping -c1 -W1 $h.mgmt; done returns 100 % success.

Typical trap. Cabling “however it comes out” with no physical labelling and no diagram. When F4 arrives and a NetworkPolicy has to be troubleshot, not knowing which physical interface carries which VLAN doubles the diagnosis time of every incident for ever.

Why F0 does not overlap with F1. Until every server has a management IP and a live BMC, OS bootstrap cannot be automated. Every hour invested in F0 saves hours in every later phase. It is the phase with the best ROI in the project and the only one that admits no shortcuts.

F1 — Bare metal: the operational base camp

What gets built. The operating system image (stable Debian or Ubuntu LTS) provisioned via PXE or cloud-init with the cloud-config versioned in git. Each node has: a coherent hostname, LVM partitions, kernel ≥ 6.6, the containerd container runtime, NVIDIA drivers on the GPU nodes, chrony synchronising against your own servers, the operator’s SSH key as the only way in, and nvidia-smi passing a smoke test on the GPU nodes.

Dependencies. F0 closed. It needs the management network working so PXE responds and so the bastion can reach every node.

Validation gate that closes F1.

ansible -i inventory all -m ping returns 100 % success (or the equivalent with Salt / Pulumi / etc).
Every GPU node passes nvidia-smi showing the expected GPUs with a consistent driver across nodes.
Each node’s clock drifts by less than 50 ms from the reference NTP.
A physical reboot of a node leaves it in exactly the same state after boot (idempotency).

Typical trap. NVIDIA drivers installed by hand with apt install or with NVIDIA’s .run script. It works on day one and breaks the day of the first kernel update. The operational rule already established in the post on the five levels: the drivers end up managed by the GPU Operator in F5; whatever is done now is only so that nvidia-smi passes the smoke test, not for production.

Possible overlap. F1 can start for some nodes while F0 is still being finished on others (real greenfield rarely delivers every server on the same day). The F1 gate is per cluster, not per individual node.

F2 — Kubernetes cluster operational

What gets built. RKE2 installed with three HA control plane nodes, all workers joined (CPU and GPU), Cilium as CNI with kubeProxyReplacement enabled and a BGP control plane pointing at the ToR switches from F0, Rook-Ceph deployed on the storage nodes to cover block (RBD), filesystem (CephFS) and object (S3-compatible RGW), kubectl get nodes returning every node Ready, a first test pod with a PVC mounting and data persisting after the pod restarts.

Dependencies. F1 closed (drivers + container runtime). Switches with BGP configured (closed in F0). NVMe disks partitioned or available raw for Ceph OSDs.

Validation gate that closes F2.

kubectl get nodes -o wide shows every node Ready with the expected Kubernetes version.
A Deployment with replicas=3 and per-node antiAffinity starts and the pods land on different nodes.
An RWO PVC (RBD) creates a volume, the pod writes data, the pod is deleted, another pod mounts it and reads the data.
An RWX PVC (CephFS) does the same with two pods writing simultaneously.
An RGW bucket via s3cmd or mc accepts put and get over TLS.
Hubble (the read side of the CNI) shows flow logs between two pods in different namespaces.
Chaos test: draining a non-GPU worker node; the workloads are rescheduled automatically.

Typical trap. Starting to kubectl apply real workloads in F2 without GitOps. The backlog of things-applied-by-hand grows faster than the capacity to migrate it to git afterwards. The rule: in F2 only the cluster prerequisites are applied (CNI, CSI, default storage class). Any application workload waits for F3.

Possible overlap. F2 ↔ F3. While the cluster is being built, the GitOps repo is prepared in parallel (directory structure, first Helm releases). When F2 closes, Flux is plugged into the repo and everything that was going to be kubectl apply is already a reconciled manifest.

F3 — GitOps and infrastructure observability

What gets built. Forgejo deployed first (it is a prerequisite for everything that follows). A gitops-infra repo with the initial structure (apps/, infrastructure/, tenants/, clusters/). Flux installed and reconciling that repo. The prerequisite workloads applied by hand in F2 are moved to the repo and reconciled (no operational kubectl apply remains). VictoriaMetrics + vmagent scraping metrics. Grafana with initial dashboards (USE/RED + cluster + Ceph + Cilium). Loki receiving logs via vector/fluent-bit. Alertmanager + Keep routing alerts to a chat channel. Barman Cloud backups for Postgres (future CNPG) and scheduled Ceph snapshots.

Dependencies. F2 closed. An RGW bucket to store backups (covered by the Ceph from F2).

Validation gate that closes F3.

A change applied to the repo is reflected in the cluster in under 5 minutes with no manual intervention.
A change applied with kubectl edit directly to the cluster is detected by Flux and reverted (binding drift detection, not merely observational).
Grafana shows dashboards for the cluster, Ceph, Cilium and GPU nodes (DCGM does not arrive until F5, but basic node metrics do).
A test alert sent to Alertmanager reaches the chat channel in under 1 minute.
Restoring a Postgres backup on a temporary cluster returns coherent data (the test defines the real RPO).

Typical trap. Having Helm charts in git but still applying them with helm install from the terminal. That is level 1 dressed up as level 2. F3 only closes when Flux is the only authority that applies changes and humans edit the repo, not the cluster.

Possible overlap. F3 ↔ F4. While F3 is being closed, the Defguard and cert-manager manifests can be prepared in the repo. When they reconcile they have somewhere to land.

F4 — Identity, certificates, secrets, policies

What gets built. Defguard deployed with its dedicated Postgres (CNPG). An initial realm with the platform operators enrolled with MFA and WireGuard. OIDC integrated into kube-apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim), into Forgejo, into Grafana, into Alertmanager, one single SSO. cert-manager installed with an internal CA issuing internal certs for mTLS and with Let’s Encrypt ACME for edge certs. SOPS configured with a KMS (it can be a physical HSM, an age key in a vault, or an external Vault) and External Secrets Operator syncing secrets into the cluster. Kyverno deployed with initial policies in audit mode for a week, then promoted to enforce. A default-deny NetworkPolicy applied to every existing namespace. Tetragon enabled for runtime security. The kube-apiserver audit log sent to Loki with long retention.

Dependencies. F3 closed (Flux applies the manifests, VM/Loki ingest metrics and logs).

Validation gate that closes F4.

kubectl with a shared admin kubeconfig stops working; every operator uses their own OIDC token with MFA.
A secret in plain data: in a commit is rejected by the pre-commit hook (or by Kyverno admission).
A pod without securityContext.runAsNonRoot=true is rejected by Kyverno at admission.
A deliberately wrong NetworkPolicy (allow-all) in a tenant namespace is rejected.
An audit of the last day returns the complete list of actors and changes (minimum regulatory footprint).
Basic internal pen-test: an attacker with a forged kubeconfig fails MFA and is recorded.

Typical trap. Kyverno in permanent audit mode because “we do not want to break production workloads”. F4 closes when the policies are in enforce. Until then, you are still in F3 wearing an F4 face.

Why F4 does not overlap with F5. F5 introduces GPU pods that move a lot of VRAM and a lot of compute. Without default-deny NetworkPolicy, without OIDC RBAC, without Kyverno blocking insecure configurations, the GPU pods are the juiciest attack surface in the cluster. Any compromise in F5 with F4 not closed is near-total access to the expensive hardware.

F5 — GPU platform with LLM-aware observability

What gets built. NVIDIA GPU Operator via Flux with the driver version decided in F1 (no longer touched by hand). DCGM Exporter exposing GPU metrics to VictoriaMetrics. MIG manager configured for the nodes where it makes sense (for example, on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs and embeddings). Topology Manager with the single-numa-node policy. KEDA with the Prometheus scaler installed and an example ScaledObject pointing at a vLLM metric (vllm:num_requests_running). OpenTelemetry Collector with OTLP receivers, attributes processors (enriching spans with tenant_id, priority_tier), exporters to Langfuse and to Tempo. LeaderWorkerSet API enabled for tensor parallel topologies. OME (Operator Model Engine) or vLLM Production Stack deployed as a controller, still with no models loaded.

Dependencies. F4 closed (the GPU pods inherit default-deny NetworkPolicy, OIDC RBAC and Kyverno policies).

Validation gate that closes F5.

A test pod requesting nvidia.com/gpu: 1 is scheduled on the right node and nvidia-smi from inside the container sees the right GPU (whole or a MIG slice).
DCGM Exporter exposes metrics in Grafana (utilisation, VRAM, temperature, NVLink bandwidth) for every GPU.
A test vLLM Deployment starts with a small model (for example, a 7B FP16) loaded from Ceph RGW.
An OpenTelemetry span generated by that vLLM reaches Langfuse with correct gen_ai.* attributes.
KEDA scales the test Deployment from 1 to N replicas under synthetic load and back to 1 when it stops.
A GPU Operator upgrade to a new version drains and reschedules the GPU pods with no loss of service.

Typical trap. Loading the large model “just to try” before DCGM and OTel are green. When something fails, there will be no metrics to distinguish OOM, thermal throttling, a driver mismatch or a network problem, and you diagnose blind. The rule: small model first, golden path green, then the large model.

Possible overlap. None with F6. F6 is atomic.

F6 — LLM stack in production

What gets built. The seven layers of the inference stack described in the corresponding post, deployed in this order:

Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets). Some components already existed from F3 as data; here they specialise for RAG with their initial collections and schemas.
Embeddings + reranker (Infinity with multilingual-e5-large, TEI with bge-reranker-v2-m3). This is the layer that must be green before any large model, because RAG depends on it.
LLM inference (vLLM Production Stack with the general LLM and the code LLM). It loads models from Ceph RGW. The initial multi-LoRA pool is empty.
Gateway (Envoy AI Gateway) with Defguard OAuth, routing by body.model, rate-limit per tenant. This is the point that opens traffic to the outside.
LLM-aware observability (Langfuse plugged into the OTel from F5).
GitOps control plane and dependency tracking were already active since F3 and F4 respectively; here you simply add the catalogue of the new LLM services to them.

Dependencies. All the previous phases closed.

Validation gate that closes F6.

A curl to the public endpoint with a Defguard bearer token receives a correct chat completion response in technical Spanish, with trace_id propagated.
The trace appears in Langfuse with complete gen_ai.* attributes, a latency breakdown and its own tenant_id.
A 5 % traffic canary to the new model over 24 h degrades neither quality nor latency metrics.
A controlled traffic burst triggers KEDA, the replicas scale, and P95 latency stays within budget.
A deliberate failure of one vLLM pod does not affect endpoint availability (replicas + reschedule).
The internal operator demonstrates the complete path for revoking a tenant’s access in under 5 minutes (Defguard → Kyverno → closing the NetworkPolicy).

Typical trap. Opening real customer traffic before having the incident runbook signed off, the SLO negotiated and the continuity plan tested. F6 is technically closed; operationally, the platform is still an experiment until the first real postmortem proves the team knows how to respond.

The maths that matter: relative weight of effort per phase

Without committing to calendar weeks, we can quantify the relative weight of the engineering effort per phase in a typical greenfield. The curve is not uniform:

$$ \text{effort}_{F_i} \approx \text{base}_i \cdot (1 + \epsilon_i) $$

where $\text{base}_i$ is the nominal effort and $\epsilon_i$ is the surprises factor (wrong cabling, incompatible drivers, badly issued certificates, version conflicts). The table below gives the nominal relative weight and the typical surprise factor observed:

Phase	Nominal weight	Typical surprise factor ε	Mean effective weight
F0 — Hardware	8 %	0.5 (1× to 2×)	12 %
F1 — Bare metal	6 %	0.3	8 %
F2 — k8s cluster	12 %	0.4	17 %
F3 — GitOps + obs	14 %	0.5	21 %
F4 — Identity + policies	18 %	0.7	31 %
F5 — GPU plane	10 %	0.4	14 %
F6 — LLM stack live	8 %	0.3	10 %
Buffer / integration	24 %	—	—

Two operational observations. F4 concentrates more surprises than any other phase (OIDC federation across four or five apps with different configurations, Kyverno policies that knock over legitimate workloads, secrets broken by poorly tested encryption). F0 has a high surprise coefficient relative to its size because any cabling or labelling error is discovered late and paid for dearly. The two practical consequences: plan F4 with generous margin and do not skimp on time in F0, because every hour saved there costs five later.

Critical path and slack. The critical path is linear F0 → F1 → F2 → F3 → F4 → F5 → F6. The only real slack is the overlaps already identified:

F2 ↔ F3 (slack ~30 %): prepare the repo and initial dashboards while the cluster is being built.
F3 ↔ F4 (slack ~20 %): identity manifests ready as F3 closes, applied immediately.
Within F4: policies in audit mode running in parallel with the Defguard setup.

Nothing shortens the critical path by more than about 15 % of the total. Anyone promising a productive greenfield in half a reasonable timescale is selling something else: probably skipping F4, or loading F6 with an F5 that is green-but-not-validated.

Final diagram: the complete deployment schedule

The schedule is not decorative: each row defines what gets built in its phase and the gate that closes it. A phase is not considered finished until its gate is green. A phase with a yellow gate drags every later one; trying to jump to the next with a partially met gate is what produces, several weeks later, the incident that forces you to “go back to F4 with production running”, the most expensive situation in the whole cost matrix of the post on the five levels.

Typical planning mistakes

Patterns that delay or sink the greenfield deployment, regardless of the tools chosen:

1. Buying the LLM before the cluster. Starting the project with “which model are we going to serve” instead of “what platform can sustain any reasonable model”. The model is an interchangeable parameter; the platform is not.

2. Underestimating F0. “The network team does that.” Yes, but the output of F0 is consumed by every later phase. If the network team delivers late, the whole project is late, and nobody had marked it as the critical path.

3. Overlapping F4 with F5 “to save time”. It is the one dependency with no slack. If you try to overlap, F5 ends up operating with policies in permanent audit (you are not in F4) or without OIDC integrated (operators with a shared kubeconfig touching GPUs). Both antipatterns stay in production.

4. Skipping the small model smoke test in F5. “Let’s go straight for the 70B.” When something fails (and something will), there will be no baseline to diagnose against.

5. Treating F6 as “turning on vLLM”. F6 includes gateway, LLM-aware observability, runbook, SLO, continuity plan. Turning on vLLM takes five minutes; closing F6 is weeks of validation and sign-off.

6. Not defining gates in writing. If the gates are not written down, they are negotiable after the fact. “This already counts as F4” is the sentence that precedes the next six months of retrofit.

7. Assigning a phase to a single owner. Every phase needs at least two people who understand it. Staff turnover on long projects destroys knowledge; written gates + cross review preserve it.

8. Forgetting the descent route. The post focuses on going up. Day-to-day operation (the descent, in the analogy) is another story that also deserves planning: runbooks, on-call, upgrade capacity, end-of-life plan. Teams that only plan the ascent reach the summit and stay there with no oxygen.

Applied to typical on-premise hardware: 4×H100 SXM

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM per GPU node, 3 control plane nodes, 3-5 CPU worker nodes, 2 GPU worker nodes), the temporal split of the work is distributed like this:

F0 (hardware)
└─ 8 physical servers racked + switches + BMC + management IPs
 ├─ 3 nodes cp-01..03 — control plane (no GPU)
 ├─ 3 nodes worker-cpu-01..03 — CPU plane (Forgejo, Ceph, observability)
 └─ 2 nodes worker-gpu-01..02 — GPU plane (4×H100 SXM each)

F1 (bare metal)
└─ OS + drivers + containerd on all 8 nodes alike
 (NVIDIA drivers only on the 2 GPU nodes, `nvidia-smi` smoke)

F2 (k8s cluster)
└─ RKE2 control plane on cp-01..03 (HA with embedded etcd)
 workers joining: 3 CPU + 2 GPU
 Ceph OSDs on the 3 CPU worker nodes
 default pools: RBD-replicated-3, CephFS-replicated-3, RGW

F3 (GitOps + obs)
└─ Forgejo + Flux + VM/Grafana/Loki + Keep on the CPU plane
 first `gitops-infra` repo reconciles what came from F2

F4 (identity)
└─ Defguard on the CPU plane (StatefulSet with Postgres CNPG)
 OIDC in kube-apiserver, Forgejo, Grafana, Alertmanager
 Kyverno as a Deployment on the control plane

F5 (GPU plane)
└─ NVIDIA GPU Operator targets the GPU workers
 MIG manager: 1st GPU MIG 7g.80gb (= passthrough), 2nd 2×3g.40gb
 OTel Collector as a DaemonSet on the GPU plane + CPU plane
 first vLLM with a 7B FP16 model green

F6 (LLM stack)
└─ The 7 layers reconcile via Flux from a second repo `gitops-llm`
 first public endpoint with Defguard OAuth
 first production customer enrolled under an SLA

The physical distribution of the cluster exploits the isolation between planes defined in F0: the control plane never touches a GPU, the CPU plane concentrates relevant state (Forgejo, Ceph, Postgres CNPG, Langfuse, Defguard) and the GPU plane specialises as much as possible. That separation, decided in F0 before the first server is installed, conditions the success of the remaining phases. It is another reminder of why F0 matters more than it looks.

What we have not covered (upcoming posts)

This post walks the ascent route to the summit. Some pieces still deserve an article of their own:

The safe descent: day-to-day operation, runbooks per component, on-call, continuous capacity planning, the cluster upgrade cycle with no downtime.
Multi-site (the second summit): how two clusters are federated with Cilium Cluster Mesh and what extra phases that introduces. F3.5 (Cluster Mesh) and F4.5 (cross-site replication) are the missing phases.
The brownfield route: what changes when there is already a cluster with workloads. The phases stay the same, but the gates are applied retroactively and every step requires migration planning.
The real calendar cost: typical ranges in weeks for a platform team of 2-3 people, split by phase, with uncertainty bands.
The handoff to operations: how the platform is handed from the deployment team to the operations team, which documents are signed, what is inherited and what is renegotiated.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook documentation — rook.io/docs
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

Sun, 31 May 2026 07:00:00 +0200

TL;DR

The post on the seven layers of the LLM inference stack took many pieces for granted: an operational Kubernetes cluster, GitOps reconciling, identities resolved, GPUs visible to the scheduler, observability capable of carrying gen_ai.*. Before vLLM makes any sense, you have to reach that starting point, and you reach it by levels. This post defines five maturity levels of the platform that lives under the LLM, from a bare metal server with Linux installed (level 0) to a cluster ready to run the inference layer (level 4) and the handoff to the previous post (level 5). Each level unlocks a concrete capability, running containers reproducibly, rebuilding the cluster from git, authenticating humans via OIDC, scheduling GPUs with MIG and DCGM metrics, demonstrating compliance with no manual intervention, and each has a validation test that decides whether you are really there or just telling yourself you are. For each level: which OSS pieces cover it in 2026 (Cilium, RKE2, Flux, cert-manager, Defguard, NVIDIA GPU Operator, KEDA, Trivy, Kyverno…), the deployment order within the level, the decisions that are expensive to skip, and the antipatterns that drop you a level when you thought you were higher. The thesis: moving up a level costs little effort if you do it in time, and a lot of refactoring if you try to skip it. LLM inference demands at least level 4; anyone trying to serve LLMs from level 1 or 2 ends up paying in night-time incidents what they saved on the platform.

You are here: the five levels at a glance

Before the detail, the ladder. Each rung adds a capability absent from the one below. The level’s test is the question whose honest answer says whether you are already there.

The levels are not interchangeable. A cluster at level 2 cannot run LLMs in production with any guarantee: technically the vLLM pod loads, but at the first night-time incident you discover there is no TLS, no identities, no alerting, no GPU metrics, and no way to know who changed what. Moving up a level after LLMs are already in production costs orders of magnitude more than moving up while the cluster is still empty.

The analogy: from the street stall to the starred restaurant

Picture the scale of a hospitality business. Level 0 is the street stall: a griddle, a gas bottle, a cook improvising. It can sell food, it works, but anything that deviates from a normal day (a health inspection, an allergic customer, an order for 200 portions) sinks the business. Level 1 is the tapas bar: a properly sized kitchen, a short repeatable menu, several shifts. The cook no longer improvises daily; he works from a written menu, even if the recipes live in the head of the boss. Level 2 is the restaurant with a set lunch menu: written procedures, fixed suppliers, stock control, an incident book. If the head cook falls ill, the second can run the service without carnage. Level 3 is the restaurant with an à la carte menu and formal service: traceability of every ingredient, allergens on the menu, health certification, contracts with suppliers, mandatory staff training. Level 4 is the kitchen specialised in a complex product (sushi, haute cuisine, artisan bakery): specific tools the normal restaurant does not need (wood-fired oven, special knives, proving chamber), calibrated processes, quality metrics. Level 5 is the restaurant with a Michelin star: the whole system works, the dish is the result of the organisation, not of one person’s talent.

The analogy holds to the end, including the most interesting detail: you can operate at any level, but the promises you can keep are different. The street stall cannot promise a consistent experience to 80 diners with a booking. The cluster at level 1 cannot promise productive multi-tenant LLM service with an SLA. In both cases the problem is not the technical capability of the last component (the griddle cooks; the pod starts); it is the organisational capability of the whole system.

Let us go level by level.

Level 0 — Chaotic: the server with Linux and nothing else

The capability it gives. Running containers with docker/podman, running binaries, connecting the server to the network. The operator can SSH in, do things, and see results.

The test for the level. “If I reinstall the server from scratch, can I leave it identical to how it was in one afternoon, using only saved notes?”. If the answer is no (because the steps are in the head of whoever built it, in .bash_history, in an out-of-date wiki), you are at level 0.

Minimum pieces to settle before moving up to level 1.

Piece	Suggested decision in 2026	Why it matters when moving up
Linux distribution	Stable Debian or Ubuntu LTS	Long, predictable support, recent kernel available
Kernel	Recent LTS (≥ 6.6) with BPF and modern schedulers	Cilium/eBPF and recent NVIDIA drivers demand it
NVIDIA drivers	The version that matches the CUDA of the LLM engine you will serve	A driver/CUDA mismatch blocks vLLM before you start
Container runtime	`containerd`	CNCF standard, integrated with RKE2/kubeadm
Root filesystem	XFS or ext4 + LVM thin pools	Snapshots, online expansion
Time synchronisation	`chrony` with your own servers	TLS, correlated logs and short certificates demand it
Management network	Dedicated VLAN, ACLs on the switch	Isolate the control plane from workload traffic
Cluster network	LACP + jumbo frames + BGP (if you are going with Cilium)	Intra-node NVLink does not rescue the service network
BMC / IPMI	Out-of-band access with TLS and MFA	Recovery when the operating system does not boot

Antipatterns that pin you at level 0.

Pet servers (with their own names, configured by hand, not replaceable).
Changes applied with vi directly over /etc/... with no commit to a repo.
Deployment with docker-compose with no healthchecks and no automatic restart.
An inventory that lives in an Excel sheet nobody updates.

Deployment order within the level. System image from PXE/cloud-init with the initial configuration (LVM, hostname, network, SSH key, chrony) → bastion/jump host bootstrap → inventory in Ansible (or an equivalent declarative tool, even if it is later replaced) → NVIDIA drivers + container runtime → smoke test (a CUDA container passes nvidia-smi). At this point, the server is ready for Kubernetes to come in.

Level 1 — Repeatable: an operational Kubernetes cluster

The capability it gives. Scheduling containers with a scheduler, network abstraction between pods, persistent volumes, workload lifecycle, manual horizontal scaling.

The test for the level. “Can I lose a node and have the workloads rescheduled without human intervention?”. If yes, you are at level 1. If not, because the pods are pinned to nodes, because there are no replicas, because the PVCs do not reattach, you are still at 0 with Kubernetes on top.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Main alternative
k8s distribution	RKE2 (CIS-hardened by default, no commercial surcharge)	k3s for very small edge, plain kubeadm for custom cases
CNI	Cilium with kube-proxy replacement, BGP, Gateway API	Calico (without BGP it does not compete with Cilium in 2026)
CSI block + filesystem + object	Rook-Ceph (RBD + CephFS + S3-compatible RGW)	OpenEBS Mayastor + Garage for small deployments
Ingress	Cilium Gateway API (better to unify with the CNI)	NGINX Ingress, Traefik
Basic certs	Self-signed bootstrap	(cert-manager comes in at level 3)
Workload handling	`kubectl apply` + Helm from the terminal	No GitOps yet
Container registry	Any internal registry (or a trusted external one) with TLS	(a managed internal registry comes in at level 2)

Antipatterns that drop you to level 0.

Services deployed with kubectl apply from one person’s terminal and with the YAML saved nowhere.
Persistent volumes with no backup policy.
A “single-node cluster” as permanent production, an architectural single point of failure.
A CNI with no NetworkPolicy available or no BGP when the network requires it.

Deployment order within the level. RKE2 installed on at least three nodes for an HA control plane → Cilium installed in kube-proxy replacement mode + BGP control plane → Rook-Ceph on at least three nodes covering block (RBD) + filesystem (CephFS) + object (S3-compatible RGW) with 3× replication or Erasure Coding depending on the pool → smoke test (a Deployment with a PVC starts, the pods reschedule when a node is cordoned, the data persists).

Level 2 — Defined: the cluster rebuilds from git

The capability it gives. The cluster state lives in a repository. Every change goes through a commit. Anyone can rebuild the cluster (or an equivalent one) from the repo and the backups. Basic observability warns when something breaks.

The test for the level. “If I lose the whole cluster, can I recreate it in X hours from the repo + the backups, with no manual intervention beyond the bootstrap?”. The hours are negotiable; what defines the level is that the repo + the backups are enough, not that the person-who-knows is available.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
Forge	Forgejo (or Gitea, GitLab CE)	Self-hosted OSS, community fork of Gitea, open governance
GitOps reconciler	Flux	CNCF graduated, native multi-tenancy, lightweight
Image registry	Forgejo Container Registry	Next to the code, no extra piece
Metrics TSDB	VictoriaMetrics + vmagent	Higher throughput than plain Prometheus, long retention, PromQL compatible
Visualisation	Grafana	The de facto standard
Logs	Loki or Vector	OSS, integrated with Grafana
Alerting	Alertmanager + Keep (OSS orchestrator)	Keep adds multi-channel routing with no lock-in
DB backups	Barman Cloud (Postgres)	The standard for CNPG
Object / dataset backups	Ceph RGW multisite + CephFS snapshots	Cross-pool and cross-site

Antipatterns that drop you to level 1.

kubectl apply run in production outside the repo (undetected drift).
main branches with write permissions for humans without review.
A monolithic repo with no tenant/infra/apps separation (cross changes that cannot be audited).
Metrics kept no longer than 7 days (no SLO observable over a month).
Alerting that fires for everything (fatigue) or for nothing (silence).

Deployment order within the level. Forgejo deployed first (it is a prerequisite for everything else) → Flux installed and pointing at the manifests repo → an initial repository with the Cilium and Rook-Ceph Helm releases reconciled by Flux (replacing the level 1 kubectl apply) → VictoriaMetrics + Grafana + Loki via Helm/Flux → Postgres backups and Ceph snapshots scheduled → smoke test (tear the cluster down, restore from repo + backup, the services come back).

Level 3 — Managed: identities, certificates, secrets and policies

The capability it gives. Every human operating the cluster does so with their own identity (no shared kubeconfig), with MFA and with limited permissions. Automatic internal TLS. Encrypted versioned secrets. Policies that reject insecure configurations before they reach the cluster. A complete audit of who did what.

The test for the level. “If an attacker gets hold of an administrator’s laptop, what can they do in production?”. At level 3 the answer is “little”: MFA blocks the second factor, Kyverno policies block destructive changes without approval, the NetworkPolicies prevent lateral movement, the secrets are encrypted with an external KMS, and the audit log remains. At level 2, “everything”.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
IdP / OIDC	Defguard	Spanish OSS, WireGuard + OIDC + 2FA, multi-org
Federation with the cluster	OIDC in kube-apiserver, OIDC in Forgejo, OIDC in Grafana	Consistent SSO
Internal PKI	cert-manager + Trust Manager	De facto standard, ACME and internal CA
External ACME	Let’s Encrypt for edge certs	No cost, automated
Secrets in git	SOPS + age or an external KMS	Versionable, encrypted in the repo
Secret syncing	External Secrets Operator	Pull from KMS / Vault into the cluster
Policy as code	Kyverno (or OPA Gatekeeper)	Kyverno has a shallower learning curve
NetworkPolicy	Cilium NetworkPolicy + L7	Default deny per namespace
Runtime security	Tetragon (Cilium)	eBPF, complements NetworkPolicy with detection
Vulnerability scanning	Trivy in the CI pipeline + admission	SBOM per image, blocking of critical CVEs
Audit log	kube-apiserver with `--audit-policy-file` sent to Loki	Regulatory traceability

Minimum Kyverno policies to keep alive.

Deny :latest images or images without a sha digest.
Deny pods without securityContext.runAsNonRoot=true.
Deny pods without resources.limits (CPU + memory).
Deny Services without an owner=<team> label.
Deny changes in critical namespaces (kube-system, flux-system) without an approval label.

Antipatterns that drop you to level 2.

A kubeconfig shared between administrators.
Secrets in plain data: in a manifest committed to the repo.
NetworkPolicy absent from new namespaces by default (implicit allow-all).
kubectl edit or kubectl patch in production without going through the repo.

Deployment order within the level. Defguard deployed and enrolled with WireGuard / OIDC → OIDC integration with kube-apiserver, Forgejo and Grafana → cert-manager installed and issuing internal certificates (your own CA for mTLS, Let’s Encrypt for the edge) → SOPS configured and External Secrets Operator installed → migration of secrets from plain to encrypted → Kyverno with initial policies in audit mode, then enforce → default-deny NetworkPolicy per namespace → Tetragon enabled → smoke test (try to bypass each policy and check that admission rejects it).

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

The capability it gives. The Kubernetes scheduler sees the GPUs, distinguishes them, can partition them (MIG) or multiplex them (time-slicing), expose DCGM metrics, autoscale with KEDA using metrics from the LLM workload itself (vllm:num_requests_running, vllm:gpu_cache_usage_perc), and carry traces with the GenAI semantic conventions. Everything needed for the LLM inference stack to rest on a platform that understands its nature.

The test for the level. “If I deploy a pod asking for nvidia.com/gpu: 1, does it get scheduled on the right GPU, with the right slice, with DCGM metrics exposed, with GenAI observability ready to receive spans?”. If yes, you are at level 4. If the answer requires “it depends which node and who deploys it”, not yet.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
GPU device plugin	NVIDIA GPU Operator	Deploys drivers, container toolkit, DCGM and MIG manager with one operator
HW partitioning	MIG (Multi-Instance GPU) on the H100 where it applies	Real hardware isolation, not time-slicing
GPU metrics	DCGM Exporter	SM utilisation, VRAM, temperature, throttling, NVLink bandwidth
LLM metrics	vLLM Prometheus endpoint + scrape	TTFT, TPOT, KV cache, prefix hit rate
Autoscaling	KEDA with a Prometheus ScaledObject	Scales on LLM metrics, not on CPU
LLM operators	vLLM Production Stack / OME (Operator Model Engine)	Declarative handling of models / adapters
Traces	OpenTelemetry Collector with OTLP receivers + processors + exporters	`gen_ai.*` semantic conventions (post)
LeaderWorkerSet	LeaderWorkerSet API (k8s 1.30+)	Tensor parallel topology coherent with NVLink
Topology Manager	enabled with `single-numa-node`	Pinning GPU pods to the right NUMA node

The key decision: MIG, time-slicing or passthrough.

MIG divides an H100 into 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb (slices with real HW isolation). Useful for serving several small models or reserving capacity per tenant with a guarantee. Limitation: up to 7 instances per GPU, predefined profiles.
Time-slicing shares a GPU between several pods with no HW isolation. Useful for dev/test, not for multi-tenant production with an SLA.
Passthrough assigns the whole GPU to one pod. Useful for tensor parallel across several GPUs on the same node (a large LLM with TP=4).

For a productive LLM platform, the practical rule: passthrough for the large models with TP, MIG for embeddings and small models that cohabit, never time-slicing in production.

Antipatterns that drop you to level 3.

Installing NVIDIA drivers by hand outside the GPU Operator (silent breakage when Kubernetes is upgraded).
Serving an LLM with requests.gpu: 1 without having decided on MIG / passthrough (you end up with idle GPUs from fragmentation or pods stepping on each other).
KEDA autoscaling on CPU (the classic HorizontalPodAutoscaler) on pods that are almost always at 10 % CPU but 95 % KV cache.
OpenTelemetry deployed but without the gen_ai.* semantic conventions (the traces are not LLM-aware).

Deployment order within the level. NVIDIA GPU Operator installed via Helm/Flux with the driver version that matches the chosen LLM engine → DCGM Exporter enabled and metrics visible in Grafana (NVIDIA dashboards imported) → MIG manager configured for the nodes where it makes sense (a typical mix on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs + embeddings) → OpenTelemetry Collector with attributes processors to enrich spans with your own labels (tenant_id, priority_tier) + exporters to Langfuse and to Tempo → KEDA installed with an example ScaledObject pointing at vllm:num_requests_running → vLLM Production Stack or OME to declare models as a CRD → smoke test (a vLLM Deployment declared via CRD starts, serves a token, exposes metrics, the trace reaches Langfuse, KEDA scales under synthetic load).

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

Once at level 4, the cluster meets the contract that the post on the seven layers assumed as its starting point. Level 5 adds no infrastructure: it adds the LLM stack proper. For completeness, the seven components of level 5 are:

Gateway (Envoy AI Gateway) — it comes first, directing traffic to LLM inference and embeddings.
LLM inference (vLLM Production Stack or OME with vLLM) — on the GPUs already discovered by the level 4 GPU Operator.
Embeddings + reranker (Infinity, TEI) — a pod separate from the LLM, already covered in the previous post.
Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets) — most of it already existed at level 2 as data; now it specialises for RAG.
LLM-aware observability (Langfuse) — it plugs into the level 4 OTel chain.
GitOps control plane — the one from level 2 remains the only legitimate authority.
Dependency tracking (Hubble flows + Otterize) — on the Cilium that already existed at level 1.

The criterion for promoting from level 4 to level 5 is not technical: it is contractual. The cluster already supports LLMs; the decision is when to open real customer traffic. Promotion demands: a green golden eval of the model, a signed incident runbook, negotiated SLOs, a continuity plan, and mapping to ENS / NIS2 / 42001 where it applies.

The maths that matter: what skipping a level costs

To quantify the thesis of the post, an order-of-magnitude estimate of the cost of moving up each level in time versus moving up after production is running. The figures are engineering time with a small platform team (2-3 people), assuming templates and prior experience.

Level	Time to build on an empty cluster	Time to retrofit with production running
0 → 1	1-2 weeks	1-2 weeks (little downstream refactoring)
1 → 2	2-3 weeks	4-8 weeks (migrating everything to git)
2 → 3	2-4 weeks	8-16 weeks (image rebuilds, secret migration, retroactive RBAC)
3 → 4	1-2 weeks	4-8 weeks (reconfiguring GPUs, moving models to MIG, instrumenting `gen_ai.*`)
4 → 5	1-2 weeks	2-4 weeks
Total 0 → 5	~10-15 weeks	~20-40 weeks if done in the wrong order

The typical multiplier observable in practice: 2× to 3× the cost if it is done in the wrong order. And that assumes it gets done at all, since many projects never get past level 2 because “the identity thing” can always wait for another sprint. By the time the incident arrives, it is too late to start.

Beyond time, the operational cost (night-time incidents, security escapes, invisible debt) grows exponentially with the gap between the actual level and the required level. A cluster at level 2 serving productive LLMs to regulated customers is a time bomb: technically it works, organisationally it does not.

Final diagram: the complete ladder with pieces

The ladder is not decorative: each level enables the next. You cannot have LLM-aware observability (level 4) without OTel deployed via Flux (level 2). You cannot have automatic internal TLS (level 3) without a root PKI that lives somewhere (registry and certificates managed from level 2). You cannot have KEDA scaling on vLLM metrics (level 4) without Prometheus / VictoriaMetrics scraping (level 2). The levels are not a conceptual hierarchy: they are a hierarchy of installation dependencies.

Typical design decisions that break progress

Mistakes seen repeatedly that drag the cluster back a level:

1. Jumping from level 1 straight to level 4. “We are in a hurry to serve the LLM, we will do the identity and GitOps stuff later.” Later is always two orders of magnitude more expensive and always arrives after the first incident.

2. Confusing Helm with GitOps. Having Helm charts is not level 2. It is level 1 with templates. Level 2 demands that a reconciler (Flux/ArgoCD) applies the charts from a repo, detects drift and warns.

3. cert-manager with no usage policy. Having auto-renewed certificates but using TLS only at the ingress, with no internal mTLS between services, leaves the TLS promise lame and turns level 3 into a cosplay of level 3.

4. NVIDIA drivers by hand. It works on day one and breaks the day of the first kernel upgrade. The rule: drivers always via the GPU Operator, never operating system packages.

5. Prometheus metrics but 7 days of retention. Without long retention (≥ 90 days) there is no honest SLO. VictoriaMetrics with a year of retention costs little more than Prometheus with 7 days, and it unlocks compliance and serious postmortems.

6. OIDC only for kube-apiserver. If Forgejo, Grafana, Defguard and vLLM each have their own auth system, you do not have SSO, you have islands. An honest level 3 demands federation.

7. Kyverno in permanent audit mode. Policies that do not reject are not policies, they are alerts. At some point you have to move to enforce. Until then, you are still at level 2 wearing a level 3 face.

8. MIG with no conscious decision on the profile. Configuring MIG with the default profile without having measured the size of the models that will cohabit leaves fragmented GPUs with slices nobody uses. The rule: MIG only if you have measured and decided the profiles in advance.

They all share one root: declaring the level without passing the level’s test. Saying “we already did GitOps” when things are still applied with kubectl edit in prod. Saying “we already did identity” when there is a shared admin kubeconfig. Saying “we are ready for LLMs” when there is no DCGM Exporter and no Langfuse plugged in.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), a reasonable setup after passing the five levels distributes the components like this:

control plane (3 nodes without GPU, hostnames cp-01..03)
├── kube-apiserver, etcd, controller-manager, scheduler
├── Flux, Forgejo, cert-manager, External Secrets, Kyverno
└── Tetragon (DaemonSet here too)

worker plane (≥ 3 nodes without GPU, hostnames worker-cpu-01..03)
├── Cilium agent (DaemonSet)
├── Rook-Ceph OSDs + MONs + MDS (CephFS) + RGW (S3)
├── VictoriaMetrics + Grafana + Loki
├── Defguard (StatefulSet)
└── Langfuse + OTel Collector

GPU worker plane (≥ 2 nodes with 4×H100 SXM, hostnames worker-gpu-01..02)
├── NVIDIA GPU Operator (driver + container toolkit)
├── DCGM Exporter (DaemonSet)
├── MIG manager (configuring the chosen profile)
├── vLLM (Deployment) — general LLM TP=4 occupies 4 GPUs (passthrough)
├── vLLM (Deployment) — code LLM TP=2 occupies 2 GPUs
├── Infinity (embeddings) — 2 replicas cohabit on 2 MIG slices
└── KEDA scaler listening to vLLM metrics

The operational rule: the control plane and the CPU plane are separated from the GPU plane. An incident on the GPU plane must not take the control plane with it (the control plane is what recovers the cluster). And the CPU plane concentrates everything that moves relevant state (Forgejo, Rook-Ceph, Postgres CNPG, Langfuse): it is the heart to protect.

The GPU hardware specialises as much as possible: GPU pods run only on GPU nodes, and GPU nodes run nothing CPU-bound beyond the operational overhead (Cilium, GPU Operator, DCGM). This is enforced with nodeSelector + taints/tolerations + a Kyverno policy that rejects pods with no GPU requests being scheduled onto GPU nodes.

What we have not covered (upcoming posts)

This post walks the vertical path upwards. Some horizontal and cross-cutting pieces still deserve an article of their own:

Multi-site active/standby: how two clusters are federated with Cilium Cluster Mesh and what changes at each level when there are two sites instead of one.
Migration between levels with real traffic: how a cluster already in production is retrofitted to the next level with no downtime.
Day-to-day operation: runbooks per level, which dashboards to look at each morning, which SLOs to define per component.
The cost plane: what each level costs in hardware, energy, engineering hours, optional OSS licences (commercial support from Rancher, Cilium Enterprise, etc.) and when each expense is justified.
Operationalised compliance: how levels 3 and 4 map to ENS High, NIS2 and ISO/IEC 42001 controls without turning the cluster into a paperwork exercise.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook-Ceph — rook.io
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
vLLM Production Stack — docs.vllm.ai/en/latest/serving/production_stack.html
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
CIS Kubernetes Benchmark
NIST SP 800-207 — Zero Trust Architecture

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

Platform on lo0 — Tech Blog

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

TL;DR

You are here: the seven phases and their dependencies

The analogy: the expedition to an eight-thousand-metre summit

F0 — Hardware in the room: base camp

F1 — Bare metal: the operational base camp

F2 — Kubernetes cluster operational

F3 — GitOps and infrastructure observability

F4 — Identity, certificates, secrets, policies

F5 — GPU platform with LLM-aware observability

F6 — LLM stack in production

The maths that matter: relative weight of effort per phase

Final diagram: the complete deployment schedule

Typical planning mistakes

Applied to typical on-premise hardware: 4×H100 SXM

What we have not covered (upcoming posts)

See also

References

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

TL;DR

You are here: the five levels at a glance

The analogy: from the street stall to the starred restaurant

Level 0 — Chaotic: the server with Linux and nothing else

Level 1 — Repeatable: an operational Kubernetes cluster

Level 2 — Defined: the cluster rebuilds from git

Level 3 — Managed: identities, certificates, secrets and policies

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

The maths that matter: what skipping a level costs

Final diagram: the complete ladder with pieces

Typical design decisions that break progress

Applied to typical on-premise hardware: a 4×H100 SXM cluster

What we have not covered (upcoming posts)

See also

References

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

TL;DR

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

The four layers of a multi-tenant inference platform

Layer 1 — AI Gateway: the single front door

Why centralise

The three dominant options in 2026

When to choose each one

The minimum the gateway has to do

Example of a multi-tenant LiteLLM configuration

Layer 2 — Policy & Quota Plane: what each tenant can do

Technical quotas

Economic budgets

Model whitelist and blacklist

Priority classes

Admission control

The typical pattern in 2026

Layer 3 — Isolation Plane: isolating workloads physically

Three NVIDIA mechanisms for sharing a GPU

The choice for multi-tenant in 2026

MIG partitions on the H100

Approach A — A shared large model with quotas at the gateway

Approach B — Dedicating GPUs per model / tenant

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

The operational choice: start with A, move up to C if you need to

Isolation at the Kubernetes level

Layer 4 — Observability Plane: seeing what happens per tenant

The four mandatory properties

Showback vs chargeback

Tools

The minimum multi-tenant dashboard

Sizing on mid-scale GPU clusters: concrete decisions

Default decisions

An example tenant scheme

When to add hardware

Common operational traps

A gateway without auth: a backdoor into the cluster

MIG and NVLink are incompatible

Quotas pinned to the cluster’s ceiling

No multi-tenant observability from day 1

Showback that never reaches chargeback

Non-whitelisted models eating the budget

Badly calibrated priority classes