Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

Contents

TL;DR

The two previous posts in this architectural trilogy settled the pieces: the seven layers of the LLM inference stack describe the components above the cluster, and the five maturity levels of the platform describe the strata below. This post settles the when: in what order each thing gets deployed when you start from zero, hardware bought, racks installed, physical cabling done, and you want to reach a cluster serving the first production token to a customer. Seven nominal phases F0 to F6 with no calendar commitments, organised by technical dependencies (you do not enter F3 without the F2 gate) and with an identifiable critical path. F0 hardware inventory and electrical/network connectivity. F1 bare metal OS + drivers + container runtime. F2 Kubernetes cluster with CNI and Ceph storage operational. F3 GitOps and infrastructure observability. F4 identity, TLS, secrets and policies. F5 GPU platform with LLM-aware observability. F6 the LLM stack operational and open to production traffic. For each phase: what gets built, what has to be ready first (dependencies between phases), the gate that validates its closure, and the typical trap that delays the critical path. The thesis: an on-premise LLM platform sinks far more often from sequencing badly than from choosing badly. The tools have all been invented; the order is the only thing every team rediscovers.

You are here: the seven phases and their dependencies

The phases do not run in pure series. F2 and F3 can be started at the same time to speed things up (installing Kubernetes and preparing the GitOps repo in parallel). F4 can overlap with the tail end of F3. F5 waits for F4 to close because GPU pods demand NetworkPolicy and RBAC from day one. F6 is a single atomic step: the cluster goes into production or it does not.

The red arrows are the critical path: the sequential bottleneck that no parallelism can shorten. The grey arrows are dependencies that admit partial overlap. Recognising where to overlap and where not is the difference between a three-month deployment and a six-month one for the same perimeter.

The analogy: the expedition to an eight-thousand-metre summit

An expedition to a high alpine summit is not a long trek. It is a series of camps that are set up in order, each with its altitude, its function and its validation gate: if you do not acclimatise properly at base camp, you cannot go up to C1 without risk; if C2 does not have its kitchen and its radio running, you cannot send people higher; if the summit push is attempted without porters in the high camps, there is no safe descent.

Greenfield deployment of an LLM platform works identically. F0 is the arrival of the material at base camp: boxes, sponsors, permits, first review. F1 is setting up an operational base camp: kitchen, tents, generator. F2 is the climb to C1: there is real altitude now (a k8s cluster running) and you breathe differently. F3 is C2: it adds communications, planning and operational acclimatisation. F4 is C3, the last night before the push: team roped up, oxygen ready, every protocol verified. F5 is the day of the summit push, intense effort, thin margins. F6 is the summit and the start of a safe descent: from here on the expedition is in day-to-day operation, no longer in construction.

The analogy holds two useful lessons: you do not skip a camp (going straight from base camp to the summit kills the team), and the gates are technical, not emotional (if the barometer says storm, you do not set out, however much enthusiasm there is). The platform team that follows those two rules reaches the summit. The one that negotiates them does not.

F0 — Hardware in the room: base camp

What gets built in this phase. An inventory of the hardware received (servers, switches, PDUs, BMC), racks assembled, electrical and data cabling terminated, physical labelling of each machine (rack/U/function), connectivity to the corporate network, management IPs assigned, BMC reachable over VPN with MFA, first ping to each node from the bastion.

Dependencies. None technical, this is the phase before the software. There are procurement dependencies (servers bought, switches bought), civil works dependencies (a room with enough cooling, a raised floor) and administrative ones (datacenter access for the technicians).

Validation gate that closes F0.

Every node appears in the inventory with (hostname, MAC, management IP, data IP, rack, U, function, owner).
Each node’s BMC responds to ipmitool power status and to the HTTPS UI from the management VPN.
The top-of-rack switch has its configuration versioned in git (even if there is no cluster GitOps yet, the switch configs are).
A command for h in $(cat hosts); do ping -c1 -W1 $h.mgmt; done returns 100 % success.

Typical trap. Cabling “however it comes out” with no physical labelling and no diagram. When F4 arrives and a NetworkPolicy has to be troubleshot, not knowing which physical interface carries which VLAN doubles the diagnosis time of every incident for ever.

Why F0 does not overlap with F1. Until every server has a management IP and a live BMC, OS bootstrap cannot be automated. Every hour invested in F0 saves hours in every later phase. It is the phase with the best ROI in the project and the only one that admits no shortcuts.

F1 — Bare metal: the operational base camp

What gets built. The operating system image (stable Debian or Ubuntu LTS) provisioned via PXE or cloud-init with the cloud-config versioned in git. Each node has: a coherent hostname, LVM partitions, kernel ≥ 6.6, the containerd container runtime, NVIDIA drivers on the GPU nodes, chrony synchronising against your own servers, the operator’s SSH key as the only way in, and nvidia-smi passing a smoke test on the GPU nodes.

Dependencies. F0 closed. It needs the management network working so PXE responds and so the bastion can reach every node.

Validation gate that closes F1.

ansible -i inventory all -m ping returns 100 % success (or the equivalent with Salt / Pulumi / etc).
Every GPU node passes nvidia-smi showing the expected GPUs with a consistent driver across nodes.
Each node’s clock drifts by less than 50 ms from the reference NTP.
A physical reboot of a node leaves it in exactly the same state after boot (idempotency).

Typical trap. NVIDIA drivers installed by hand with apt install or with NVIDIA’s .run script. It works on day one and breaks the day of the first kernel update. The operational rule already established in the post on the five levels: the drivers end up managed by the GPU Operator in F5; whatever is done now is only so that nvidia-smi passes the smoke test, not for production.

Possible overlap. F1 can start for some nodes while F0 is still being finished on others (real greenfield rarely delivers every server on the same day). The F1 gate is per cluster, not per individual node.

F2 — Kubernetes cluster operational

What gets built. RKE2 installed with three HA control plane nodes, all workers joined (CPU and GPU), Cilium as CNI with kubeProxyReplacement enabled and a BGP control plane pointing at the ToR switches from F0, Rook-Ceph deployed on the storage nodes to cover block (RBD), filesystem (CephFS) and object (S3-compatible RGW), kubectl get nodes returning every node Ready, a first test pod with a PVC mounting and data persisting after the pod restarts.

Dependencies. F1 closed (drivers + container runtime). Switches with BGP configured (closed in F0). NVMe disks partitioned or available raw for Ceph OSDs.

Validation gate that closes F2.

kubectl get nodes -o wide shows every node Ready with the expected Kubernetes version.
A Deployment with replicas=3 and per-node antiAffinity starts and the pods land on different nodes.
An RWO PVC (RBD) creates a volume, the pod writes data, the pod is deleted, another pod mounts it and reads the data.
An RWX PVC (CephFS) does the same with two pods writing simultaneously.
An RGW bucket via s3cmd or mc accepts put and get over TLS.
Hubble (the read side of the CNI) shows flow logs between two pods in different namespaces.
Chaos test: draining a non-GPU worker node; the workloads are rescheduled automatically.

Typical trap. Starting to kubectl apply real workloads in F2 without GitOps. The backlog of things-applied-by-hand grows faster than the capacity to migrate it to git afterwards. The rule: in F2 only the cluster prerequisites are applied (CNI, CSI, default storage class). Any application workload waits for F3.

Possible overlap. F2 ↔ F3. While the cluster is being built, the GitOps repo is prepared in parallel (directory structure, first Helm releases). When F2 closes, Flux is plugged into the repo and everything that was going to be kubectl apply is already a reconciled manifest.

F3 — GitOps and infrastructure observability

What gets built. Forgejo deployed first (it is a prerequisite for everything that follows). A gitops-infra repo with the initial structure (apps/, infrastructure/, tenants/, clusters/). Flux installed and reconciling that repo. The prerequisite workloads applied by hand in F2 are moved to the repo and reconciled (no operational kubectl apply remains). VictoriaMetrics + vmagent scraping metrics. Grafana with initial dashboards (USE/RED + cluster + Ceph + Cilium). Loki receiving logs via vector/fluent-bit. Alertmanager + Keep routing alerts to a chat channel. Barman Cloud backups for Postgres (future CNPG) and scheduled Ceph snapshots.

Dependencies. F2 closed. An RGW bucket to store backups (covered by the Ceph from F2).

Validation gate that closes F3.

A change applied to the repo is reflected in the cluster in under 5 minutes with no manual intervention.
A change applied with kubectl edit directly to the cluster is detected by Flux and reverted (binding drift detection, not merely observational).
Grafana shows dashboards for the cluster, Ceph, Cilium and GPU nodes (DCGM does not arrive until F5, but basic node metrics do).
A test alert sent to Alertmanager reaches the chat channel in under 1 minute.
Restoring a Postgres backup on a temporary cluster returns coherent data (the test defines the real RPO).

Typical trap. Having Helm charts in git but still applying them with helm install from the terminal. That is level 1 dressed up as level 2. F3 only closes when Flux is the only authority that applies changes and humans edit the repo, not the cluster.

Possible overlap. F3 ↔ F4. While F3 is being closed, the Defguard and cert-manager manifests can be prepared in the repo. When they reconcile they have somewhere to land.

F4 — Identity, certificates, secrets, policies

What gets built. Defguard deployed with its dedicated Postgres (CNPG). An initial realm with the platform operators enrolled with MFA and WireGuard. OIDC integrated into kube-apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim), into Forgejo, into Grafana, into Alertmanager, one single SSO. cert-manager installed with an internal CA issuing internal certs for mTLS and with Let’s Encrypt ACME for edge certs. SOPS configured with a KMS (it can be a physical HSM, an age key in a vault, or an external Vault) and External Secrets Operator syncing secrets into the cluster. Kyverno deployed with initial policies in audit mode for a week, then promoted to enforce. A default-deny NetworkPolicy applied to every existing namespace. Tetragon enabled for runtime security. The kube-apiserver audit log sent to Loki with long retention.

Dependencies. F3 closed (Flux applies the manifests, VM/Loki ingest metrics and logs).

Validation gate that closes F4.

kubectl with a shared admin kubeconfig stops working; every operator uses their own OIDC token with MFA.
A secret in plain data: in a commit is rejected by the pre-commit hook (or by Kyverno admission).
A pod without securityContext.runAsNonRoot=true is rejected by Kyverno at admission.
A deliberately wrong NetworkPolicy (allow-all) in a tenant namespace is rejected.
An audit of the last day returns the complete list of actors and changes (minimum regulatory footprint).
Basic internal pen-test: an attacker with a forged kubeconfig fails MFA and is recorded.

Typical trap. Kyverno in permanent audit mode because “we do not want to break production workloads”. F4 closes when the policies are in enforce. Until then, you are still in F3 wearing an F4 face.

Why F4 does not overlap with F5. F5 introduces GPU pods that move a lot of VRAM and a lot of compute. Without default-deny NetworkPolicy, without OIDC RBAC, without Kyverno blocking insecure configurations, the GPU pods are the juiciest attack surface in the cluster. Any compromise in F5 with F4 not closed is near-total access to the expensive hardware.

F5 — GPU platform with LLM-aware observability

What gets built. NVIDIA GPU Operator via Flux with the driver version decided in F1 (no longer touched by hand). DCGM Exporter exposing GPU metrics to VictoriaMetrics. MIG manager configured for the nodes where it makes sense (for example, on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs and embeddings). Topology Manager with the single-numa-node policy. KEDA with the Prometheus scaler installed and an example ScaledObject pointing at a vLLM metric (vllm:num_requests_running). OpenTelemetry Collector with OTLP receivers, attributes processors (enriching spans with tenant_id, priority_tier), exporters to Langfuse and to Tempo. LeaderWorkerSet API enabled for tensor parallel topologies. OME (Operator Model Engine) or vLLM Production Stack deployed as a controller, still with no models loaded.

Dependencies. F4 closed (the GPU pods inherit default-deny NetworkPolicy, OIDC RBAC and Kyverno policies).

Validation gate that closes F5.

A test pod requesting nvidia.com/gpu: 1 is scheduled on the right node and nvidia-smi from inside the container sees the right GPU (whole or a MIG slice).
DCGM Exporter exposes metrics in Grafana (utilisation, VRAM, temperature, NVLink bandwidth) for every GPU.
A test vLLM Deployment starts with a small model (for example, a 7B FP16) loaded from Ceph RGW.
An OpenTelemetry span generated by that vLLM reaches Langfuse with correct gen_ai.* attributes.
KEDA scales the test Deployment from 1 to N replicas under synthetic load and back to 1 when it stops.
A GPU Operator upgrade to a new version drains and reschedules the GPU pods with no loss of service.

Typical trap. Loading the large model “just to try” before DCGM and OTel are green. When something fails, there will be no metrics to distinguish OOM, thermal throttling, a driver mismatch or a network problem, and you diagnose blind. The rule: small model first, golden path green, then the large model.

Possible overlap. None with F6. F6 is atomic.

F6 — LLM stack in production

What gets built. The seven layers of the inference stack described in the corresponding post, deployed in this order:

Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets). Some components already existed from F3 as data; here they specialise for RAG with their initial collections and schemas.
Embeddings + reranker (Infinity with multilingual-e5-large, TEI with bge-reranker-v2-m3). This is the layer that must be green before any large model, because RAG depends on it.
LLM inference (vLLM Production Stack with the general LLM and the code LLM). It loads models from Ceph RGW. The initial multi-LoRA pool is empty.
Gateway (Envoy AI Gateway) with Defguard OAuth, routing by body.model, rate-limit per tenant. This is the point that opens traffic to the outside.
LLM-aware observability (Langfuse plugged into the OTel from F5).
GitOps control plane and dependency tracking were already active since F3 and F4 respectively; here you simply add the catalogue of the new LLM services to them.

Dependencies. All the previous phases closed.

Validation gate that closes F6.

A curl to the public endpoint with a Defguard bearer token receives a correct chat completion response in technical Spanish, with trace_id propagated.
The trace appears in Langfuse with complete gen_ai.* attributes, a latency breakdown and its own tenant_id.
A 5 % traffic canary to the new model over 24 h degrades neither quality nor latency metrics.
A controlled traffic burst triggers KEDA, the replicas scale, and P95 latency stays within budget.
A deliberate failure of one vLLM pod does not affect endpoint availability (replicas + reschedule).
The internal operator demonstrates the complete path for revoking a tenant’s access in under 5 minutes (Defguard → Kyverno → closing the NetworkPolicy).

Typical trap. Opening real customer traffic before having the incident runbook signed off, the SLO negotiated and the continuity plan tested. F6 is technically closed; operationally, the platform is still an experiment until the first real postmortem proves the team knows how to respond.

The maths that matter: relative weight of effort per phase

Without committing to calendar weeks, we can quantify the relative weight of the engineering effort per phase in a typical greenfield. The curve is not uniform:

$$ \text{effort}_{F_i} \approx \text{base}_i \cdot (1 + \epsilon_i) $$

where $\text{base}_i$ is the nominal effort and $\epsilon_i$ is the surprises factor (wrong cabling, incompatible drivers, badly issued certificates, version conflicts). The table below gives the nominal relative weight and the typical surprise factor observed:

Phase	Nominal weight	Typical surprise factor ε	Mean effective weight
F0 — Hardware	8 %	0.5 (1× to 2×)	12 %
F1 — Bare metal	6 %	0.3	8 %
F2 — k8s cluster	12 %	0.4	17 %
F3 — GitOps + obs	14 %	0.5	21 %
F4 — Identity + policies	18 %	0.7	31 %
F5 — GPU plane	10 %	0.4	14 %
F6 — LLM stack live	8 %	0.3	10 %
Buffer / integration	24 %	—	—

Two operational observations. F4 concentrates more surprises than any other phase (OIDC federation across four or five apps with different configurations, Kyverno policies that knock over legitimate workloads, secrets broken by poorly tested encryption). F0 has a high surprise coefficient relative to its size because any cabling or labelling error is discovered late and paid for dearly. The two practical consequences: plan F4 with generous margin and do not skimp on time in F0, because every hour saved there costs five later.

Critical path and slack. The critical path is linear F0 → F1 → F2 → F3 → F4 → F5 → F6. The only real slack is the overlaps already identified:

F2 ↔ F3 (slack ~30 %): prepare the repo and initial dashboards while the cluster is being built.
F3 ↔ F4 (slack ~20 %): identity manifests ready as F3 closes, applied immediately.
Within F4: policies in audit mode running in parallel with the Defguard setup.

Nothing shortens the critical path by more than about 15 % of the total. Anyone promising a productive greenfield in half a reasonable timescale is selling something else: probably skipping F4, or loading F6 with an F5 that is green-but-not-validated.

Final diagram: the complete deployment schedule

The schedule is not decorative: each row defines what gets built in its phase and the gate that closes it. A phase is not considered finished until its gate is green. A phase with a yellow gate drags every later one; trying to jump to the next with a partially met gate is what produces, several weeks later, the incident that forces you to “go back to F4 with production running”, the most expensive situation in the whole cost matrix of the post on the five levels.

Typical planning mistakes

Patterns that delay or sink the greenfield deployment, regardless of the tools chosen:

1. Buying the LLM before the cluster. Starting the project with “which model are we going to serve” instead of “what platform can sustain any reasonable model”. The model is an interchangeable parameter; the platform is not.

2. Underestimating F0. “The network team does that.” Yes, but the output of F0 is consumed by every later phase. If the network team delivers late, the whole project is late, and nobody had marked it as the critical path.

3. Overlapping F4 with F5 “to save time”. It is the one dependency with no slack. If you try to overlap, F5 ends up operating with policies in permanent audit (you are not in F4) or without OIDC integrated (operators with a shared kubeconfig touching GPUs). Both antipatterns stay in production.

4. Skipping the small model smoke test in F5. “Let’s go straight for the 70B.” When something fails (and something will), there will be no baseline to diagnose against.

5. Treating F6 as “turning on vLLM”. F6 includes gateway, LLM-aware observability, runbook, SLO, continuity plan. Turning on vLLM takes five minutes; closing F6 is weeks of validation and sign-off.

6. Not defining gates in writing. If the gates are not written down, they are negotiable after the fact. “This already counts as F4” is the sentence that precedes the next six months of retrofit.

7. Assigning a phase to a single owner. Every phase needs at least two people who understand it. Staff turnover on long projects destroys knowledge; written gates + cross review preserve it.

8. Forgetting the descent route. The post focuses on going up. Day-to-day operation (the descent, in the analogy) is another story that also deserves planning: runbooks, on-call, upgrade capacity, end-of-life plan. Teams that only plan the ascent reach the summit and stay there with no oxygen.

Applied to typical on-premise hardware: 4×H100 SXM

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM per GPU node, 3 control plane nodes, 3-5 CPU worker nodes, 2 GPU worker nodes), the temporal split of the work is distributed like this:

F0 (hardware)
└─ 8 physical servers racked + switches + BMC + management IPs
   ├─ 3 nodes cp-01..03   — control plane (no GPU)
   ├─ 3 nodes worker-cpu-01..03 — CPU plane (Forgejo, Ceph, observability)
   └─ 2 nodes worker-gpu-01..02 — GPU plane (4×H100 SXM each)

F1 (bare metal)
└─ OS + drivers + containerd on all 8 nodes alike
   (NVIDIA drivers only on the 2 GPU nodes, `nvidia-smi` smoke)

F2 (k8s cluster)
└─ RKE2 control plane on cp-01..03 (HA with embedded etcd)
   workers joining: 3 CPU + 2 GPU
   Ceph OSDs on the 3 CPU worker nodes
   default pools: RBD-replicated-3, CephFS-replicated-3, RGW

F3 (GitOps + obs)
└─ Forgejo + Flux + VM/Grafana/Loki + Keep on the CPU plane
   first `gitops-infra` repo reconciles what came from F2

F4 (identity)
└─ Defguard on the CPU plane (StatefulSet with Postgres CNPG)
   OIDC in kube-apiserver, Forgejo, Grafana, Alertmanager
   Kyverno as a Deployment on the control plane

F5 (GPU plane)
└─ NVIDIA GPU Operator targets the GPU workers
   MIG manager: 1st GPU MIG 7g.80gb (= passthrough), 2nd 2×3g.40gb
   OTel Collector as a DaemonSet on the GPU plane + CPU plane
   first vLLM with a 7B FP16 model green

F6 (LLM stack)
└─ The 7 layers reconcile via Flux from a second repo `gitops-llm`
   first public endpoint with Defguard OAuth
   first production customer enrolled under an SLA

The physical distribution of the cluster exploits the isolation between planes defined in F0: the control plane never touches a GPU, the CPU plane concentrates relevant state (Forgejo, Ceph, Postgres CNPG, Langfuse, Defguard) and the GPU plane specialises as much as possible. That separation, decided in F0 before the first server is installed, conditions the success of the remaining phases. It is another reminder of why F0 matters more than it looks.

What we have not covered (upcoming posts)

This post walks the ascent route to the summit. Some pieces still deserve an article of their own:

The safe descent: day-to-day operation, runbooks per component, on-call, continuous capacity planning, the cluster upgrade cycle with no downtime.
Multi-site (the second summit): how two clusters are federated with Cilium Cluster Mesh and what extra phases that introduces. F3.5 (Cluster Mesh) and F4.5 (cross-site replication) are the missing phases.
The brownfield route: what changes when there is already a cluster with workloads. The phases stay the same, but the gates are applied retroactively and every step requires migration planning.
The real calendar cost: typical ranges in weeks for a platform team of 2-3 people, split by phase, with uncertainty bands.
The handoff to operations: how the platform is handed from the deployment team to the operations team, which documents are signed, what is inherited and what is renegotiated.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook documentation — rook.io/docs
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai