Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

TL;DR

The post on the seven layers of the LLM inference stack took many pieces for granted: an operational Kubernetes cluster, GitOps reconciling, identities resolved, GPUs visible to the scheduler, observability capable of carrying gen_ai.*. Before vLLM makes any sense, you have to reach that starting point, and you reach it by levels. This post defines five maturity levels of the platform that lives under the LLM, from a bare metal server with Linux installed (level 0) to a cluster ready to run the inference layer (level 4) and the handoff to the previous post (level 5). Each level unlocks a concrete capability, running containers reproducibly, rebuilding the cluster from git, authenticating humans via OIDC, scheduling GPUs with MIG and DCGM metrics, demonstrating compliance with no manual intervention, and each has a validation test that decides whether you are really there or just telling yourself you are. For each level: which OSS pieces cover it in 2026 (Cilium, RKE2, Flux, cert-manager, Defguard, NVIDIA GPU Operator, KEDA, Trivy, Kyverno…), the deployment order within the level, the decisions that are expensive to skip, and the antipatterns that drop you a level when you thought you were higher. The thesis: moving up a level costs little effort if you do it in time, and a lot of refactoring if you try to skip it. LLM inference demands at least level 4; anyone trying to serve LLMs from level 1 or 2 ends up paying in night-time incidents what they saved on the platform.

You are here: the five levels at a glance

Before the detail, the ladder. Each rung adds a capability absent from the one below. The level’s test is the question whose honest answer says whether you are already there.

The levels are not interchangeable. A cluster at level 2 cannot run LLMs in production with any guarantee: technically the vLLM pod loads, but at the first night-time incident you discover there is no TLS, no identities, no alerting, no GPU metrics, and no way to know who changed what. Moving up a level after LLMs are already in production costs orders of magnitude more than moving up while the cluster is still empty.

The analogy: from the street stall to the starred restaurant

Picture the scale of a hospitality business. Level 0 is the street stall: a griddle, a gas bottle, a cook improvising. It can sell food, it works, but anything that deviates from a normal day (a health inspection, an allergic customer, an order for 200 portions) sinks the business. Level 1 is the tapas bar: a properly sized kitchen, a short repeatable menu, several shifts. The cook no longer improvises daily; he works from a written menu, even if the recipes live in the head of the boss. Level 2 is the restaurant with a set lunch menu: written procedures, fixed suppliers, stock control, an incident book. If the head cook falls ill, the second can run the service without carnage. Level 3 is the restaurant with an à la carte menu and formal service: traceability of every ingredient, allergens on the menu, health certification, contracts with suppliers, mandatory staff training. Level 4 is the kitchen specialised in a complex product (sushi, haute cuisine, artisan bakery): specific tools the normal restaurant does not need (wood-fired oven, special knives, proving chamber), calibrated processes, quality metrics. Level 5 is the restaurant with a Michelin star: the whole system works, the dish is the result of the organisation, not of one person’s talent.

The analogy holds to the end, including the most interesting detail: you can operate at any level, but the promises you can keep are different. The street stall cannot promise a consistent experience to 80 diners with a booking. The cluster at level 1 cannot promise productive multi-tenant LLM service with an SLA. In both cases the problem is not the technical capability of the last component (the griddle cooks; the pod starts); it is the organisational capability of the whole system.

Let us go level by level.

Level 0 — Chaotic: the server with Linux and nothing else

The capability it gives. Running containers with docker/podman, running binaries, connecting the server to the network. The operator can SSH in, do things, and see results.

The test for the level. “If I reinstall the server from scratch, can I leave it identical to how it was in one afternoon, using only saved notes?”. If the answer is no (because the steps are in the head of whoever built it, in .bash_history, in an out-of-date wiki), you are at level 0.

Minimum pieces to settle before moving up to level 1.

Antipatterns that pin you at level 0.

• Pet servers (with their own names, configured by hand, not replaceable).
• Changes applied with vi directly over /etc/... with no commit to a repo.
• Deployment with docker-compose with no healthchecks and no automatic restart.
• An inventory that lives in an Excel sheet nobody updates.

Deployment order within the level. System image from PXE/cloud-init with the initial configuration (LVM, hostname, network, SSH key, chrony) → bastion/jump host bootstrap → inventory in Ansible (or an equivalent declarative tool, even if it is later replaced) → NVIDIA drivers + container runtime → smoke test (a CUDA container passes nvidia-smi). At this point, the server is ready for Kubernetes to come in.

Level 1 — Repeatable: an operational Kubernetes cluster

The capability it gives. Scheduling containers with a scheduler, network abstraction between pods, persistent volumes, workload lifecycle, manual horizontal scaling.

The test for the level. “Can I lose a node and have the workloads rescheduled without human intervention?”. If yes, you are at level 1. If not, because the pods are pinned to nodes, because there are no replicas, because the PVCs do not reattach, you are still at 0 with Kubernetes on top.

Minimum pieces for the level.

Antipatterns that drop you to level 0.

• Services deployed with kubectl apply from one person’s terminal and with the YAML saved nowhere.
• Persistent volumes with no backup policy.
• A “single-node cluster” as permanent production, an architectural single point of failure.
• A CNI with no NetworkPolicy available or no BGP when the network requires it.

Deployment order within the level. RKE2 installed on at least three nodes for an HA control plane → Cilium installed in kube-proxy replacement mode + BGP control plane → Rook-Ceph on at least three nodes covering block (RBD) + filesystem (CephFS) + object (S3-compatible RGW) with 3× replication or Erasure Coding depending on the pool → smoke test (a Deployment with a PVC starts, the pods reschedule when a node is cordoned, the data persists).

Level 2 — Defined: the cluster rebuilds from git

The capability it gives. The cluster state lives in a repository. Every change goes through a commit. Anyone can rebuild the cluster (or an equivalent one) from the repo and the backups. Basic observability warns when something breaks.

The test for the level. “If I lose the whole cluster, can I recreate it in X hours from the repo + the backups, with no manual intervention beyond the bootstrap?”. The hours are negotiable; what defines the level is that the repo + the backups are enough, not that the person-who-knows is available.

Minimum pieces for the level.

Antipatterns that drop you to level 1.

• kubectl apply run in production outside the repo (undetected drift).
• main branches with write permissions for humans without review.
• A monolithic repo with no tenant/infra/apps separation (cross changes that cannot be audited).
• Metrics kept no longer than 7 days (no SLO observable over a month).
• Alerting that fires for everything (fatigue) or for nothing (silence).

Deployment order within the level. Forgejo deployed first (it is a prerequisite for everything else) → Flux installed and pointing at the manifests repo → an initial repository with the Cilium and Rook-Ceph Helm releases reconciled by Flux (replacing the level 1 kubectl apply) → VictoriaMetrics + Grafana + Loki via Helm/Flux → Postgres backups and Ceph snapshots scheduled → smoke test (tear the cluster down, restore from repo + backup, the services come back).

Level 3 — Managed: identities, certificates, secrets and policies

The capability it gives. Every human operating the cluster does so with their own identity (no shared kubeconfig), with MFA and with limited permissions. Automatic internal TLS. Encrypted versioned secrets. Policies that reject insecure configurations before they reach the cluster. A complete audit of who did what.

The test for the level. “If an attacker gets hold of an administrator’s laptop, what can they do in production?”. At level 3 the answer is “little”: MFA blocks the second factor, Kyverno policies block destructive changes without approval, the NetworkPolicies prevent lateral movement, the secrets are encrypted with an external KMS, and the audit log remains. At level 2, “everything”.

Minimum pieces for the level.

Minimum Kyverno policies to keep alive.

• Deny :latest images or images without a sha digest.
• Deny pods without securityContext.runAsNonRoot=true.
• Deny pods without resources.limits (CPU + memory).
• Deny Services without an owner=<team> label.
• Deny changes in critical namespaces (kube-system, flux-system) without an approval label.

Antipatterns that drop you to level 2.

• A kubeconfig shared between administrators.
• Secrets in plain data: in a manifest committed to the repo.
• NetworkPolicy absent from new namespaces by default (implicit allow-all).
• kubectl edit or kubectl patch in production without going through the repo.

Deployment order within the level. Defguard deployed and enrolled with WireGuard / OIDC → OIDC integration with kube-apiserver, Forgejo and Grafana → cert-manager installed and issuing internal certificates (your own CA for mTLS, Let’s Encrypt for the edge) → SOPS configured and External Secrets Operator installed → migration of secrets from plain to encrypted → Kyverno with initial policies in audit mode, then enforce → default-deny NetworkPolicy per namespace → Tetragon enabled → smoke test (try to bypass each policy and check that admission rejects it).

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

The capability it gives. The Kubernetes scheduler sees the GPUs, distinguishes them, can partition them (MIG) or multiplex them (time-slicing), expose DCGM metrics, autoscale with KEDA using metrics from the LLM workload itself (vllm:num_requests_running, vllm:gpu_cache_usage_perc), and carry traces with the GenAI semantic conventions. Everything needed for the LLM inference stack to rest on a platform that understands its nature.

The test for the level. “If I deploy a pod asking for nvidia.com/gpu: 1, does it get scheduled on the right GPU, with the right slice, with DCGM metrics exposed, with GenAI observability ready to receive spans?”. If yes, you are at level 4. If the answer requires “it depends which node and who deploys it”, not yet.

Minimum pieces for the level.

The key decision: MIG, time-slicing or passthrough.

• MIG divides an H100 into 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb (slices with real HW isolation). Useful for serving several small models or reserving capacity per tenant with a guarantee. Limitation: up to 7 instances per GPU, predefined profiles.
• Time-slicing shares a GPU between several pods with no HW isolation. Useful for dev/test, not for multi-tenant production with an SLA.
• Passthrough assigns the whole GPU to one pod. Useful for tensor parallel across several GPUs on the same node (a large LLM with TP=4).

For a productive LLM platform, the practical rule: passthrough for the large models with TP, MIG for embeddings and small models that cohabit, never time-slicing in production.

Antipatterns that drop you to level 3.

• Installing NVIDIA drivers by hand outside the GPU Operator (silent breakage when Kubernetes is upgraded).
• Serving an LLM with requests.gpu: 1 without having decided on MIG / passthrough (you end up with idle GPUs from fragmentation or pods stepping on each other).
• KEDA autoscaling on CPU (the classic HorizontalPodAutoscaler) on pods that are almost always at 10 % CPU but 95 % KV cache.
• OpenTelemetry deployed but without the gen_ai.* semantic conventions (the traces are not LLM-aware).

Deployment order within the level. NVIDIA GPU Operator installed via Helm/Flux with the driver version that matches the chosen LLM engine → DCGM Exporter enabled and metrics visible in Grafana (NVIDIA dashboards imported) → MIG manager configured for the nodes where it makes sense (a typical mix on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs + embeddings) → OpenTelemetry Collector with attributes processors to enrich spans with your own labels (tenant_id, priority_tier) + exporters to Langfuse and to Tempo → KEDA installed with an example ScaledObject pointing at vllm:num_requests_running → vLLM Production Stack or OME to declare models as a CRD → smoke test (a vLLM Deployment declared via CRD starts, serves a token, exposes metrics, the trace reaches Langfuse, KEDA scales under synthetic load).

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

Once at level 4, the cluster meets the contract that the post on the seven layers assumed as its starting point. Level 5 adds no infrastructure: it adds the LLM stack proper. For completeness, the seven components of level 5 are:

1. Gateway (Envoy AI Gateway) — it comes first, directing traffic to LLM inference and embeddings.
2. LLM inference (vLLM Production Stack or OME with vLLM) — on the GPUs already discovered by the level 4 GPU Operator.
3. Embeddings + reranker (Infinity, TEI) — a pod separate from the LLM, already covered in the previous post.
4. Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets) — most of it already existed at level 2 as data; now it specialises for RAG.
5. LLM-aware observability (Langfuse) — it plugs into the level 4 OTel chain.
6. GitOps control plane — the one from level 2 remains the only legitimate authority.
7. Dependency tracking (Hubble flows + Otterize) — on the Cilium that already existed at level 1.

The criterion for promoting from level 4 to level 5 is not technical: it is contractual. The cluster already supports LLMs; the decision is when to open real customer traffic. Promotion demands: a green golden eval of the model, a signed incident runbook, negotiated SLOs, a continuity plan, and mapping to ENS / NIS2 / 42001 where it applies.

The maths that matter: what skipping a level costs

To quantify the thesis of the post, an order-of-magnitude estimate of the cost of moving up each level in time versus moving up after production is running. The figures are engineering time with a small platform team (2-3 people), assuming templates and prior experience.

The typical multiplier observable in practice: 2× to 3× the cost if it is done in the wrong order. And that assumes it gets done at all, since many projects never get past level 2 because “the identity thing” can always wait for another sprint. By the time the incident arrives, it is too late to start.

Beyond time, the operational cost (night-time incidents, security escapes, invisible debt) grows exponentially with the gap between the actual level and the required level. A cluster at level 2 serving productive LLMs to regulated customers is a time bomb: technically it works, organisationally it does not.

Final diagram: the complete ladder with pieces

The ladder is not decorative: each level enables the next. You cannot have LLM-aware observability (level 4) without OTel deployed via Flux (level 2). You cannot have automatic internal TLS (level 3) without a root PKI that lives somewhere (registry and certificates managed from level 2). You cannot have KEDA scaling on vLLM metrics (level 4) without Prometheus / VictoriaMetrics scraping (level 2). The levels are not a conceptual hierarchy: they are a hierarchy of installation dependencies.

Typical design decisions that break progress

Mistakes seen repeatedly that drag the cluster back a level:

1. Jumping from level 1 straight to level 4. “We are in a hurry to serve the LLM, we will do the identity and GitOps stuff later.” Later is always two orders of magnitude more expensive and always arrives after the first incident.

2. Confusing Helm with GitOps. Having Helm charts is not level 2. It is level 1 with templates. Level 2 demands that a reconciler (Flux/ArgoCD) applies the charts from a repo, detects drift and warns.

3. cert-manager with no usage policy. Having auto-renewed certificates but using TLS only at the ingress, with no internal mTLS between services, leaves the TLS promise lame and turns level 3 into a cosplay of level 3.

4. NVIDIA drivers by hand. It works on day one and breaks the day of the first kernel upgrade. The rule: drivers always via the GPU Operator, never operating system packages.

5. Prometheus metrics but 7 days of retention. Without long retention (≥ 90 days) there is no honest SLO. VictoriaMetrics with a year of retention costs little more than Prometheus with 7 days, and it unlocks compliance and serious postmortems.

6. OIDC only for kube-apiserver. If Forgejo, Grafana, Defguard and vLLM each have their own auth system, you do not have SSO, you have islands. An honest level 3 demands federation.

7. Kyverno in permanent audit mode. Policies that do not reject are not policies, they are alerts. At some point you have to move to enforce. Until then, you are still at level 2 wearing a level 3 face.

8. MIG with no conscious decision on the profile. Configuring MIG with the default profile without having measured the size of the models that will cohabit leaves fragmented GPUs with slices nobody uses. The rule: MIG only if you have measured and decided the profiles in advance.

They all share one root: declaring the level without passing the level’s test. Saying “we already did GitOps” when things are still applied with kubectl edit in prod. Saying “we already did identity” when there is a shared admin kubeconfig. Saying “we are ready for LLMs” when there is no DCGM Exporter and no Langfuse plugged in.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), a reasonable setup after passing the five levels distributes the components like this:

control plane (3 nodes without GPU, hostnames cp-01..03)
├── kube-apiserver, etcd, controller-manager, scheduler
├── Flux, Forgejo, cert-manager, External Secrets, Kyverno
└── Tetragon (DaemonSet here too)

worker plane (≥ 3 nodes without GPU, hostnames worker-cpu-01..03)
├── Cilium agent (DaemonSet)
├── Rook-Ceph OSDs + MONs + MDS (CephFS) + RGW (S3)
├── VictoriaMetrics + Grafana + Loki
├── Defguard (StatefulSet)
└── Langfuse + OTel Collector

GPU worker plane (≥ 2 nodes with 4×H100 SXM, hostnames worker-gpu-01..02)
├── NVIDIA GPU Operator (driver + container toolkit)
├── DCGM Exporter (DaemonSet)
├── MIG manager (configuring the chosen profile)
├── vLLM (Deployment) — general LLM TP=4 occupies 4 GPUs (passthrough)
├── vLLM (Deployment) — code LLM TP=2 occupies 2 GPUs
├── Infinity (embeddings) — 2 replicas cohabit on 2 MIG slices
└── KEDA scaler listening to vLLM metrics

The operational rule: the control plane and the CPU plane are separated from the GPU plane. An incident on the GPU plane must not take the control plane with it (the control plane is what recovers the cluster). And the CPU plane concentrates everything that moves relevant state (Forgejo, Rook-Ceph, Postgres CNPG, Langfuse): it is the heart to protect.

The GPU hardware specialises as much as possible: GPU pods run only on GPU nodes, and GPU nodes run nothing CPU-bound beyond the operational overhead (Cilium, GPU Operator, DCGM). This is enforced with nodeSelector + taints/tolerations + a Kyverno policy that rejects pods with no GPU requests being scheduled onto GPU nodes.

What we have not covered (upcoming posts)

This post walks the vertical path upwards. Some horizontal and cross-cutting pieces still deserve an article of their own:

• Multi-site active/standby: how two clusters are federated with Cilium Cluster Mesh and what changes at each level when there are two sites instead of one.
• Migration between levels with real traffic: how a cluster already in production is retrofitted to the next level with no downtime.
• Day-to-day operation: runbooks per level, which dashboards to look at each morning, which SLOs to define per component.
• The cost plane: what each level costs in hardware, energy, engineering hours, optional OSS licences (commercial support from Rancher, Cilium Enterprise, etc.) and when each expense is justified.
• Operationalised compliance: how levels 3 and 4 map to ENS High, NIS2 and ISO/IEC 42001 controls without turning the cluster into a paperwork exercise.

See also

• GitOps for the inference stack with Flux: operating the assistant as code — the maturity level at which the platform is operated as code with Flux.

• Anatomy of an on-premise LLM inference stack — what gets built on top of a level 4 cluster. This post is its architectural prequel.

• The OSS catalogue for LLMOps in six stages — individual cards for many of the pieces cited here.

• LLM tracing with OpenTelemetry GenAI — the OTel of level 4 with detail on the gen_ai.* semantic conventions.

• The six-stage LLMOps pipeline — the operational framework that lives above a level 5 cluster.

• GPU observability for LLM inference — the twelve DCGM and vLLM metrics that turn level 4 into a real cockpit.

• LLM autoscaling on Kubernetes — KEDA with custom metrics is the piece that activates level 4’s “OPTIMISED FOR GPU” when the first traffic peak arrives.

• Canary, blue-green and shadow for LLM models — progressive rollout is the first thing level 5 must be able to do without the operator having to watch the screen.

• The LLM inference router — the piece that appears from level 3 onwards (it needs level 3’s OIDC + cert-manager + NetworkPolicy to make sense) and that connects level 4’s pools to external clients.

References

• RKE2 Documentation — docs.rke2.io
• Cilium documentation — docs.cilium.io
• Rook-Ceph — rook.io
• Flux GitOps toolkit — fluxcd.io
• Forgejo — forgejo.org
• cert-manager — cert-manager.io
• External Secrets Operator — external-secrets.io
• Kyverno — kyverno.io
• NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
• DCGM Exporter — github.com/NVIDIA/dcgm-exporter
• KEDA — keda.sh
• LeaderWorkerSet API — github.com/kubernetes-sigs/lws
• vLLM Production Stack — docs.vllm.ai/en/latest/serving/production_stack.html
• OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
• CIS Kubernetes Benchmark
• NIST SP 800-207 — Zero Trust Architecture