Architecture on lo0 — Tech Blog

One big replica or many small ones: the decision that defines your platform

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

With 4 GPUs available you have two basic options: one vLLM instance using all 4 (TP=4) or two independent instances using 2 each (TP=2 × 2 replicas). The first gives lower latency per individual request. The second gives higher aggregate throughput at high concurrency, better fault tolerance and finer-grained scaling. The crossover point, where the second overtakes the first, is typically between 16 and 64 concurrent requests for 70B models, much earlier than most people assume. The metric that decides it: goodput, the tokens generated within the latency SLO divided by the total.

The analogy

Two ways to organise a translation service: one senior translator with access to four specialised dictionaries at once (able to resolve any complex query in 30 seconds), or two junior translators with two dictionaries each (they take 45 seconds per complex query, but they can handle two at the same time).

For a client who arrives alone and expects a fast answer, the senior wins. For a queue of twenty clients arriving at once, the two juniors process twice as many queries per hour even though each one takes longer. The question is not who is better, but what kind of traffic you have.

The two architectures in vLLM

Architecture A: TP=4, one replica

# A single instance uses all 4 GPUs via tensor parallelism
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --gpu-memory-utilization 0.92 \
 --port 8000

GPU-0 ─┐
GPU-1 ─┤─ vLLM instance 0 ──► port 8000
GPU-2 ─┤ (TP=4, the model is
GPU-3 ─┘ split across 4 GPUs)

Every attention and FFN operation is split across 4 GPUs. They require an all-reduce after each layer (on NVLink: ~50-200 µs; on PCIe: ~2-8 ms). The complete model is available in the aggregate VRAM.

Architecture B: TP=2 × 2 replicas

# Two independent instances, each with 2 GPUs
# Instance 0 on GPU 0-1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8000

# Instance 1 on GPU 2-3
CUDA_VISIBLE_DEVICES=2,3 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8001

GPU-0 ─┐ ┌─► port 8000
GPU-1 ─┘─ vLLM instance 0 ─┘
← load balancer
GPU-2 ─┐ ┌─► port 8001
GPU-3 ─┘─ vLLM instance 1 ─┘

Each instance holds half the model. Requests are distributed across instances. No communication between instances (they are completely independent).

Why TP=4 has higher individual latency than TP=2

Tensor parallelism splits every transformer layer. After computing its fraction, each GPU needs to synchronise with the others via all-reduce before moving on to the next layer. The cost of that synchronisation:

$$\text{overhead TP} = n\_layers \times 2 \times \text{allreduce latency}$$

For Llama 3 70B (80 layers) on 4×H100 NVLink:

$$\text{overhead TP4} = 80 \times 2 \times 100\,\mu s = 16\,ms$$

On PCIe (without direct NVLink between GPUs):

$$\text{overhead TP4 PCIe} = 80 \times 2 \times 3\,ms = 480\,ms$$

That overhead is added to every decode step. With TP=2:

$$\text{overhead TP2} = 80 \times 2 \times 60\,\mu s = 9.6\,ms \text{ (NVLink)}$$

The difference between TP=2 and TP=4 on NVLink is ~6 ms per decode step, which matters for TPOT (inter-token latency) in streaming applications.

On PCIe without direct NVLink, TP=4 can be 400 ms slower per step than TP=2. For an output of 200 tokens, that is 80 extra seconds. In that scenario TP=4 over PCIe should never be used unless the model does not fit in 2 GPUs.

The crossover point: when TP=2×2 beats TP=4×1

For a 70B model on 4×H100 SXM (NVLink), aggregate throughput in tokens per second:

Concurrency | TP=4 × 1 instance | TP=2 × 2 instances | Winner
──────────────┼──────────────────────┼───────────────────────┼────────
1 | 200 tok/s | 170 tok/s | TP=4 (latency)
4 | 650 tok/s | 620 tok/s | TP=4 (slight)
16 | 1,800 tok/s | 2,100 tok/s | TP=2×2
32 | 2,400 tok/s | 3,600 tok/s | TP=2×2 (+50%)
64 | 2,800 tok/s | 5,200 tok/s | TP=2×2 (+86%)
128 | 2,900 tok/s | 5,800 tok/s | TP=2×2 (+100%)

Why they diverge at high concurrency: with TP=4, the scheduler of a single instance manages every request but the KV cache is shared. With TP=2×2, each instance has its own scheduler and KV cache: less contention, more real parallelism.

The crossover point on NVLink sits around 16-32 simultaneous requests for 70B. For smaller models (14B, 7B) the crossover happens earlier, because the TP communication overhead weighs more in relative terms.

The three implications nobody mentions

1. Fault tolerance

With TP=4 × 1 replica: if one GPU fails, the whole instance goes down. The service drops to 0% until the GPU recovers or the pod restarts on another node.

With TP=2 × 2 replicas: if one GPU fails, one instance goes down. The service keeps running at 50% capacity. For ENS/NIS2, where availability is a contractual requirement, this difference is decisive.

2. Autoscaling granularity

With KEDA or HPA based on vllm:num_waiting_seqs, autoscaling has to provision in multiples of the deploy unit:

TP=4 × 1: each new node requires 4 GPUs. The minimum scaling granularity is 4 GPUs.
TP=2 × 2: each new pod requires 2 GPUs. The minimum granularity is 2 GPUs, which is finer and more cost-efficient.

3. Quality degradation under load

TP=4 with many concurrent requests starts to suffer preemptions when the KV cache fills up. TP=2×2 spreads that pressure across two independent KV cache pools, so the probability of preemption is lower under the same total load.

Measuring the crossover point with OTel

Goodput is the right metric for comparing the two architectures. Not raw throughput (which ignores the SLO), but the tokens generated within the agreed TPOT SLO:

# Goodput: tokens generated with TPOT inside the SLO (e.g. <50ms/token)
# For TP=4:
rate(vllm:generation_tokens_total{instance="tp4"}[5m])
 * (1 - histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket{instance="tp4"}[5m])) > 0.050)

# For TP=2×2 (sum of the two instances):
sum(rate(vllm:generation_tokens_total{instance=~"tp2-.*"}[5m]))
 * (1 - histogram_quantile(0.95, sum(rate(vllm:time_per_output_token_seconds_bucket{instance=~"tp2-.*"}[5m]))) > 0.050)

A direct comparison on the same dashboard, with synthetic traffic at different concurrency levels, determines the exact crossover point for your hardware and model.

The decision by workload profile

Profile	Recommended architecture	Reason
Single-user chatbot / low concurrency (<10 simultaneous)	TP=4 × 1	Lower p50 latency, better streaming experience
Enterprise API (20-100 concurrent)	TP=2 × 2	Higher goodput, fault tolerance, finer autoscaling
Batch processing (throughput > latency)	TP=2 × 2 (or more replicas)	Maximum throughput always lies in replicas
Very large model (>80B, does not fit in 2 GPUs)	TP=4 × 1	No structural alternative
ENS / contractual availability	TP=2 × 2 minimum	Losing one GPU is not catastrophic

Kubernetes configuration for both architectures

# Parallel deployments for an A/B test or different topologies

# TP=2 instances (2 replicas per deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-tp2
spec:
 replicas: 2
 template:
 spec:
 containers:
 - name: vllm
 args: ["serve", "meta-llama/Meta-Llama-3.1-70B-Instruct",
 "--tensor-parallel-size", "2",
 "--gpu-memory-utilization", "0.92"]
 resources:
 limits:
 nvidia.com/gpu: "2" # 2 GPUs per pod
---
# Service with load balancing across the 2 replicas
apiVersion: v1
kind: Service
metadata:
 name: vllm-tp2
spec:
 selector:
 app: vllm-tp2
 ports:
 - port: 8000
 sessionAffinity: ClientIP  # for prefix cache awareness
 sessionAffinityConfig:
 clientIP:
 timeoutSeconds: 10800 # 3 hours of affinity per session

The sessionAffinity: ClientIP setting in the Kubernetes Service is the simplest way to implement session-affinity routing: requests from the same client always go to the same replica, maximising the prefix cache hit rate on the conversation history.

References

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

Sun, 31 May 2026 08:00:00 +0200

TL;DR

The two previous posts in this architectural trilogy settled the pieces: the seven layers of the LLM inference stack describe the components above the cluster, and the five maturity levels of the platform describe the strata below. This post settles the when: in what order each thing gets deployed when you start from zero, hardware bought, racks installed, physical cabling done, and you want to reach a cluster serving the first production token to a customer. Seven nominal phases F0 to F6 with no calendar commitments, organised by technical dependencies (you do not enter F3 without the F2 gate) and with an identifiable critical path. F0 hardware inventory and electrical/network connectivity. F1 bare metal OS + drivers + container runtime. F2 Kubernetes cluster with CNI and Ceph storage operational. F3 GitOps and infrastructure observability. F4 identity, TLS, secrets and policies. F5 GPU platform with LLM-aware observability. F6 the LLM stack operational and open to production traffic. For each phase: what gets built, what has to be ready first (dependencies between phases), the gate that validates its closure, and the typical trap that delays the critical path. The thesis: an on-premise LLM platform sinks far more often from sequencing badly than from choosing badly. The tools have all been invented; the order is the only thing every team rediscovers.

You are here: the seven phases and their dependencies

The phases do not run in pure series. F2 and F3 can be started at the same time to speed things up (installing Kubernetes and preparing the GitOps repo in parallel). F4 can overlap with the tail end of F3. F5 waits for F4 to close because GPU pods demand NetworkPolicy and RBAC from day one. F6 is a single atomic step: the cluster goes into production or it does not.

The red arrows are the critical path: the sequential bottleneck that no parallelism can shorten. The grey arrows are dependencies that admit partial overlap. Recognising where to overlap and where not is the difference between a three-month deployment and a six-month one for the same perimeter.

The analogy: the expedition to an eight-thousand-metre summit

An expedition to a high alpine summit is not a long trek. It is a series of camps that are set up in order, each with its altitude, its function and its validation gate: if you do not acclimatise properly at base camp, you cannot go up to C1 without risk; if C2 does not have its kitchen and its radio running, you cannot send people higher; if the summit push is attempted without porters in the high camps, there is no safe descent.

Greenfield deployment of an LLM platform works identically. F0 is the arrival of the material at base camp: boxes, sponsors, permits, first review. F1 is setting up an operational base camp: kitchen, tents, generator. F2 is the climb to C1: there is real altitude now (a k8s cluster running) and you breathe differently. F3 is C2: it adds communications, planning and operational acclimatisation. F4 is C3, the last night before the push: team roped up, oxygen ready, every protocol verified. F5 is the day of the summit push, intense effort, thin margins. F6 is the summit and the start of a safe descent: from here on the expedition is in day-to-day operation, no longer in construction.

The analogy holds two useful lessons: you do not skip a camp (going straight from base camp to the summit kills the team), and the gates are technical, not emotional (if the barometer says storm, you do not set out, however much enthusiasm there is). The platform team that follows those two rules reaches the summit. The one that negotiates them does not.

F0 — Hardware in the room: base camp

What gets built in this phase. An inventory of the hardware received (servers, switches, PDUs, BMC), racks assembled, electrical and data cabling terminated, physical labelling of each machine (rack/U/function), connectivity to the corporate network, management IPs assigned, BMC reachable over VPN with MFA, first ping to each node from the bastion.

Dependencies. None technical, this is the phase before the software. There are procurement dependencies (servers bought, switches bought), civil works dependencies (a room with enough cooling, a raised floor) and administrative ones (datacenter access for the technicians).

Validation gate that closes F0.

Every node appears in the inventory with (hostname, MAC, management IP, data IP, rack, U, function, owner).
Each node’s BMC responds to ipmitool power status and to the HTTPS UI from the management VPN.
The top-of-rack switch has its configuration versioned in git (even if there is no cluster GitOps yet, the switch configs are).
A command for h in $(cat hosts); do ping -c1 -W1 $h.mgmt; done returns 100 % success.

Typical trap. Cabling “however it comes out” with no physical labelling and no diagram. When F4 arrives and a NetworkPolicy has to be troubleshot, not knowing which physical interface carries which VLAN doubles the diagnosis time of every incident for ever.

Why F0 does not overlap with F1. Until every server has a management IP and a live BMC, OS bootstrap cannot be automated. Every hour invested in F0 saves hours in every later phase. It is the phase with the best ROI in the project and the only one that admits no shortcuts.

F1 — Bare metal: the operational base camp

What gets built. The operating system image (stable Debian or Ubuntu LTS) provisioned via PXE or cloud-init with the cloud-config versioned in git. Each node has: a coherent hostname, LVM partitions, kernel ≥ 6.6, the containerd container runtime, NVIDIA drivers on the GPU nodes, chrony synchronising against your own servers, the operator’s SSH key as the only way in, and nvidia-smi passing a smoke test on the GPU nodes.

Dependencies. F0 closed. It needs the management network working so PXE responds and so the bastion can reach every node.

Validation gate that closes F1.

ansible -i inventory all -m ping returns 100 % success (or the equivalent with Salt / Pulumi / etc).
Every GPU node passes nvidia-smi showing the expected GPUs with a consistent driver across nodes.
Each node’s clock drifts by less than 50 ms from the reference NTP.
A physical reboot of a node leaves it in exactly the same state after boot (idempotency).

Typical trap. NVIDIA drivers installed by hand with apt install or with NVIDIA’s .run script. It works on day one and breaks the day of the first kernel update. The operational rule already established in the post on the five levels: the drivers end up managed by the GPU Operator in F5; whatever is done now is only so that nvidia-smi passes the smoke test, not for production.

Possible overlap. F1 can start for some nodes while F0 is still being finished on others (real greenfield rarely delivers every server on the same day). The F1 gate is per cluster, not per individual node.

F2 — Kubernetes cluster operational

What gets built. RKE2 installed with three HA control plane nodes, all workers joined (CPU and GPU), Cilium as CNI with kubeProxyReplacement enabled and a BGP control plane pointing at the ToR switches from F0, Rook-Ceph deployed on the storage nodes to cover block (RBD), filesystem (CephFS) and object (S3-compatible RGW), kubectl get nodes returning every node Ready, a first test pod with a PVC mounting and data persisting after the pod restarts.

Dependencies. F1 closed (drivers + container runtime). Switches with BGP configured (closed in F0). NVMe disks partitioned or available raw for Ceph OSDs.

Validation gate that closes F2.

kubectl get nodes -o wide shows every node Ready with the expected Kubernetes version.
A Deployment with replicas=3 and per-node antiAffinity starts and the pods land on different nodes.
An RWO PVC (RBD) creates a volume, the pod writes data, the pod is deleted, another pod mounts it and reads the data.
An RWX PVC (CephFS) does the same with two pods writing simultaneously.
An RGW bucket via s3cmd or mc accepts put and get over TLS.
Hubble (the read side of the CNI) shows flow logs between two pods in different namespaces.
Chaos test: draining a non-GPU worker node; the workloads are rescheduled automatically.

Typical trap. Starting to kubectl apply real workloads in F2 without GitOps. The backlog of things-applied-by-hand grows faster than the capacity to migrate it to git afterwards. The rule: in F2 only the cluster prerequisites are applied (CNI, CSI, default storage class). Any application workload waits for F3.

Possible overlap. F2 ↔ F3. While the cluster is being built, the GitOps repo is prepared in parallel (directory structure, first Helm releases). When F2 closes, Flux is plugged into the repo and everything that was going to be kubectl apply is already a reconciled manifest.

F3 — GitOps and infrastructure observability

What gets built. Forgejo deployed first (it is a prerequisite for everything that follows). A gitops-infra repo with the initial structure (apps/, infrastructure/, tenants/, clusters/). Flux installed and reconciling that repo. The prerequisite workloads applied by hand in F2 are moved to the repo and reconciled (no operational kubectl apply remains). VictoriaMetrics + vmagent scraping metrics. Grafana with initial dashboards (USE/RED + cluster + Ceph + Cilium). Loki receiving logs via vector/fluent-bit. Alertmanager + Keep routing alerts to a chat channel. Barman Cloud backups for Postgres (future CNPG) and scheduled Ceph snapshots.

Dependencies. F2 closed. An RGW bucket to store backups (covered by the Ceph from F2).

Validation gate that closes F3.

A change applied to the repo is reflected in the cluster in under 5 minutes with no manual intervention.
A change applied with kubectl edit directly to the cluster is detected by Flux and reverted (binding drift detection, not merely observational).
Grafana shows dashboards for the cluster, Ceph, Cilium and GPU nodes (DCGM does not arrive until F5, but basic node metrics do).
A test alert sent to Alertmanager reaches the chat channel in under 1 minute.
Restoring a Postgres backup on a temporary cluster returns coherent data (the test defines the real RPO).

Typical trap. Having Helm charts in git but still applying them with helm install from the terminal. That is level 1 dressed up as level 2. F3 only closes when Flux is the only authority that applies changes and humans edit the repo, not the cluster.

Possible overlap. F3 ↔ F4. While F3 is being closed, the Defguard and cert-manager manifests can be prepared in the repo. When they reconcile they have somewhere to land.

F4 — Identity, certificates, secrets, policies

What gets built. Defguard deployed with its dedicated Postgres (CNPG). An initial realm with the platform operators enrolled with MFA and WireGuard. OIDC integrated into kube-apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim), into Forgejo, into Grafana, into Alertmanager, one single SSO. cert-manager installed with an internal CA issuing internal certs for mTLS and with Let’s Encrypt ACME for edge certs. SOPS configured with a KMS (it can be a physical HSM, an age key in a vault, or an external Vault) and External Secrets Operator syncing secrets into the cluster. Kyverno deployed with initial policies in audit mode for a week, then promoted to enforce. A default-deny NetworkPolicy applied to every existing namespace. Tetragon enabled for runtime security. The kube-apiserver audit log sent to Loki with long retention.

Dependencies. F3 closed (Flux applies the manifests, VM/Loki ingest metrics and logs).

Validation gate that closes F4.

kubectl with a shared admin kubeconfig stops working; every operator uses their own OIDC token with MFA.
A secret in plain data: in a commit is rejected by the pre-commit hook (or by Kyverno admission).
A pod without securityContext.runAsNonRoot=true is rejected by Kyverno at admission.
A deliberately wrong NetworkPolicy (allow-all) in a tenant namespace is rejected.
An audit of the last day returns the complete list of actors and changes (minimum regulatory footprint).
Basic internal pen-test: an attacker with a forged kubeconfig fails MFA and is recorded.

Typical trap. Kyverno in permanent audit mode because “we do not want to break production workloads”. F4 closes when the policies are in enforce. Until then, you are still in F3 wearing an F4 face.

Why F4 does not overlap with F5. F5 introduces GPU pods that move a lot of VRAM and a lot of compute. Without default-deny NetworkPolicy, without OIDC RBAC, without Kyverno blocking insecure configurations, the GPU pods are the juiciest attack surface in the cluster. Any compromise in F5 with F4 not closed is near-total access to the expensive hardware.

F5 — GPU platform with LLM-aware observability

What gets built. NVIDIA GPU Operator via Flux with the driver version decided in F1 (no longer touched by hand). DCGM Exporter exposing GPU metrics to VictoriaMetrics. MIG manager configured for the nodes where it makes sense (for example, on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs and embeddings). Topology Manager with the single-numa-node policy. KEDA with the Prometheus scaler installed and an example ScaledObject pointing at a vLLM metric (vllm:num_requests_running). OpenTelemetry Collector with OTLP receivers, attributes processors (enriching spans with tenant_id, priority_tier), exporters to Langfuse and to Tempo. LeaderWorkerSet API enabled for tensor parallel topologies. OME (Operator Model Engine) or vLLM Production Stack deployed as a controller, still with no models loaded.

Dependencies. F4 closed (the GPU pods inherit default-deny NetworkPolicy, OIDC RBAC and Kyverno policies).

Validation gate that closes F5.

A test pod requesting nvidia.com/gpu: 1 is scheduled on the right node and nvidia-smi from inside the container sees the right GPU (whole or a MIG slice).
DCGM Exporter exposes metrics in Grafana (utilisation, VRAM, temperature, NVLink bandwidth) for every GPU.
A test vLLM Deployment starts with a small model (for example, a 7B FP16) loaded from Ceph RGW.
An OpenTelemetry span generated by that vLLM reaches Langfuse with correct gen_ai.* attributes.
KEDA scales the test Deployment from 1 to N replicas under synthetic load and back to 1 when it stops.
A GPU Operator upgrade to a new version drains and reschedules the GPU pods with no loss of service.

Typical trap. Loading the large model “just to try” before DCGM and OTel are green. When something fails, there will be no metrics to distinguish OOM, thermal throttling, a driver mismatch or a network problem, and you diagnose blind. The rule: small model first, golden path green, then the large model.

Possible overlap. None with F6. F6 is atomic.

F6 — LLM stack in production

What gets built. The seven layers of the inference stack described in the corresponding post, deployed in this order:

Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets). Some components already existed from F3 as data; here they specialise for RAG with their initial collections and schemas.
Embeddings + reranker (Infinity with multilingual-e5-large, TEI with bge-reranker-v2-m3). This is the layer that must be green before any large model, because RAG depends on it.
LLM inference (vLLM Production Stack with the general LLM and the code LLM). It loads models from Ceph RGW. The initial multi-LoRA pool is empty.
Gateway (Envoy AI Gateway) with Defguard OAuth, routing by body.model, rate-limit per tenant. This is the point that opens traffic to the outside.
LLM-aware observability (Langfuse plugged into the OTel from F5).
GitOps control plane and dependency tracking were already active since F3 and F4 respectively; here you simply add the catalogue of the new LLM services to them.

Dependencies. All the previous phases closed.

Validation gate that closes F6.

A curl to the public endpoint with a Defguard bearer token receives a correct chat completion response in technical Spanish, with trace_id propagated.
The trace appears in Langfuse with complete gen_ai.* attributes, a latency breakdown and its own tenant_id.
A 5 % traffic canary to the new model over 24 h degrades neither quality nor latency metrics.
A controlled traffic burst triggers KEDA, the replicas scale, and P95 latency stays within budget.
A deliberate failure of one vLLM pod does not affect endpoint availability (replicas + reschedule).
The internal operator demonstrates the complete path for revoking a tenant’s access in under 5 minutes (Defguard → Kyverno → closing the NetworkPolicy).

Typical trap. Opening real customer traffic before having the incident runbook signed off, the SLO negotiated and the continuity plan tested. F6 is technically closed; operationally, the platform is still an experiment until the first real postmortem proves the team knows how to respond.

The maths that matter: relative weight of effort per phase

Without committing to calendar weeks, we can quantify the relative weight of the engineering effort per phase in a typical greenfield. The curve is not uniform:

$$ \text{effort}_{F_i} \approx \text{base}_i \cdot (1 + \epsilon_i) $$

where $\text{base}_i$ is the nominal effort and $\epsilon_i$ is the surprises factor (wrong cabling, incompatible drivers, badly issued certificates, version conflicts). The table below gives the nominal relative weight and the typical surprise factor observed:

Phase	Nominal weight	Typical surprise factor ε	Mean effective weight
F0 — Hardware	8 %	0.5 (1× to 2×)	12 %
F1 — Bare metal	6 %	0.3	8 %
F2 — k8s cluster	12 %	0.4	17 %
F3 — GitOps + obs	14 %	0.5	21 %
F4 — Identity + policies	18 %	0.7	31 %
F5 — GPU plane	10 %	0.4	14 %
F6 — LLM stack live	8 %	0.3	10 %
Buffer / integration	24 %	—	—

Two operational observations. F4 concentrates more surprises than any other phase (OIDC federation across four or five apps with different configurations, Kyverno policies that knock over legitimate workloads, secrets broken by poorly tested encryption). F0 has a high surprise coefficient relative to its size because any cabling or labelling error is discovered late and paid for dearly. The two practical consequences: plan F4 with generous margin and do not skimp on time in F0, because every hour saved there costs five later.

Critical path and slack. The critical path is linear F0 → F1 → F2 → F3 → F4 → F5 → F6. The only real slack is the overlaps already identified:

F2 ↔ F3 (slack ~30 %): prepare the repo and initial dashboards while the cluster is being built.
F3 ↔ F4 (slack ~20 %): identity manifests ready as F3 closes, applied immediately.
Within F4: policies in audit mode running in parallel with the Defguard setup.

Nothing shortens the critical path by more than about 15 % of the total. Anyone promising a productive greenfield in half a reasonable timescale is selling something else: probably skipping F4, or loading F6 with an F5 that is green-but-not-validated.

Final diagram: the complete deployment schedule

The schedule is not decorative: each row defines what gets built in its phase and the gate that closes it. A phase is not considered finished until its gate is green. A phase with a yellow gate drags every later one; trying to jump to the next with a partially met gate is what produces, several weeks later, the incident that forces you to “go back to F4 with production running”, the most expensive situation in the whole cost matrix of the post on the five levels.

Typical planning mistakes

Patterns that delay or sink the greenfield deployment, regardless of the tools chosen:

1. Buying the LLM before the cluster. Starting the project with “which model are we going to serve” instead of “what platform can sustain any reasonable model”. The model is an interchangeable parameter; the platform is not.

2. Underestimating F0. “The network team does that.” Yes, but the output of F0 is consumed by every later phase. If the network team delivers late, the whole project is late, and nobody had marked it as the critical path.

3. Overlapping F4 with F5 “to save time”. It is the one dependency with no slack. If you try to overlap, F5 ends up operating with policies in permanent audit (you are not in F4) or without OIDC integrated (operators with a shared kubeconfig touching GPUs). Both antipatterns stay in production.

4. Skipping the small model smoke test in F5. “Let’s go straight for the 70B.” When something fails (and something will), there will be no baseline to diagnose against.

5. Treating F6 as “turning on vLLM”. F6 includes gateway, LLM-aware observability, runbook, SLO, continuity plan. Turning on vLLM takes five minutes; closing F6 is weeks of validation and sign-off.

6. Not defining gates in writing. If the gates are not written down, they are negotiable after the fact. “This already counts as F4” is the sentence that precedes the next six months of retrofit.

7. Assigning a phase to a single owner. Every phase needs at least two people who understand it. Staff turnover on long projects destroys knowledge; written gates + cross review preserve it.

8. Forgetting the descent route. The post focuses on going up. Day-to-day operation (the descent, in the analogy) is another story that also deserves planning: runbooks, on-call, upgrade capacity, end-of-life plan. Teams that only plan the ascent reach the summit and stay there with no oxygen.

Applied to typical on-premise hardware: 4×H100 SXM

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM per GPU node, 3 control plane nodes, 3-5 CPU worker nodes, 2 GPU worker nodes), the temporal split of the work is distributed like this:

F0 (hardware)
└─ 8 physical servers racked + switches + BMC + management IPs
 ├─ 3 nodes cp-01..03 — control plane (no GPU)
 ├─ 3 nodes worker-cpu-01..03 — CPU plane (Forgejo, Ceph, observability)
 └─ 2 nodes worker-gpu-01..02 — GPU plane (4×H100 SXM each)

F1 (bare metal)
└─ OS + drivers + containerd on all 8 nodes alike
 (NVIDIA drivers only on the 2 GPU nodes, `nvidia-smi` smoke)

F2 (k8s cluster)
└─ RKE2 control plane on cp-01..03 (HA with embedded etcd)
 workers joining: 3 CPU + 2 GPU
 Ceph OSDs on the 3 CPU worker nodes
 default pools: RBD-replicated-3, CephFS-replicated-3, RGW

F3 (GitOps + obs)
└─ Forgejo + Flux + VM/Grafana/Loki + Keep on the CPU plane
 first `gitops-infra` repo reconciles what came from F2

F4 (identity)
└─ Defguard on the CPU plane (StatefulSet with Postgres CNPG)
 OIDC in kube-apiserver, Forgejo, Grafana, Alertmanager
 Kyverno as a Deployment on the control plane

F5 (GPU plane)
└─ NVIDIA GPU Operator targets the GPU workers
 MIG manager: 1st GPU MIG 7g.80gb (= passthrough), 2nd 2×3g.40gb
 OTel Collector as a DaemonSet on the GPU plane + CPU plane
 first vLLM with a 7B FP16 model green

F6 (LLM stack)
└─ The 7 layers reconcile via Flux from a second repo `gitops-llm`
 first public endpoint with Defguard OAuth
 first production customer enrolled under an SLA

The physical distribution of the cluster exploits the isolation between planes defined in F0: the control plane never touches a GPU, the CPU plane concentrates relevant state (Forgejo, Ceph, Postgres CNPG, Langfuse, Defguard) and the GPU plane specialises as much as possible. That separation, decided in F0 before the first server is installed, conditions the success of the remaining phases. It is another reminder of why F0 matters more than it looks.

What we have not covered (upcoming posts)

This post walks the ascent route to the summit. Some pieces still deserve an article of their own:

The safe descent: day-to-day operation, runbooks per component, on-call, continuous capacity planning, the cluster upgrade cycle with no downtime.
Multi-site (the second summit): how two clusters are federated with Cilium Cluster Mesh and what extra phases that introduces. F3.5 (Cluster Mesh) and F4.5 (cross-site replication) are the missing phases.
The brownfield route: what changes when there is already a cluster with workloads. The phases stay the same, but the gates are applied retroactively and every step requires migration planning.
The real calendar cost: typical ranges in weeks for a platform team of 2-3 people, split by phase, with uncertainty bands.
The handoff to operations: how the platform is handed from the deployment team to the operations team, which documents are signed, what is inherited and what is renegotiated.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook documentation — rook.io/docs
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

Sun, 31 May 2026 07:00:00 +0200

TL;DR

The post on the seven layers of the LLM inference stack took many pieces for granted: an operational Kubernetes cluster, GitOps reconciling, identities resolved, GPUs visible to the scheduler, observability capable of carrying gen_ai.*. Before vLLM makes any sense, you have to reach that starting point, and you reach it by levels. This post defines five maturity levels of the platform that lives under the LLM, from a bare metal server with Linux installed (level 0) to a cluster ready to run the inference layer (level 4) and the handoff to the previous post (level 5). Each level unlocks a concrete capability, running containers reproducibly, rebuilding the cluster from git, authenticating humans via OIDC, scheduling GPUs with MIG and DCGM metrics, demonstrating compliance with no manual intervention, and each has a validation test that decides whether you are really there or just telling yourself you are. For each level: which OSS pieces cover it in 2026 (Cilium, RKE2, Flux, cert-manager, Defguard, NVIDIA GPU Operator, KEDA, Trivy, Kyverno…), the deployment order within the level, the decisions that are expensive to skip, and the antipatterns that drop you a level when you thought you were higher. The thesis: moving up a level costs little effort if you do it in time, and a lot of refactoring if you try to skip it. LLM inference demands at least level 4; anyone trying to serve LLMs from level 1 or 2 ends up paying in night-time incidents what they saved on the platform.

You are here: the five levels at a glance

Before the detail, the ladder. Each rung adds a capability absent from the one below. The level’s test is the question whose honest answer says whether you are already there.

The levels are not interchangeable. A cluster at level 2 cannot run LLMs in production with any guarantee: technically the vLLM pod loads, but at the first night-time incident you discover there is no TLS, no identities, no alerting, no GPU metrics, and no way to know who changed what. Moving up a level after LLMs are already in production costs orders of magnitude more than moving up while the cluster is still empty.

The analogy: from the street stall to the starred restaurant

Picture the scale of a hospitality business. Level 0 is the street stall: a griddle, a gas bottle, a cook improvising. It can sell food, it works, but anything that deviates from a normal day (a health inspection, an allergic customer, an order for 200 portions) sinks the business. Level 1 is the tapas bar: a properly sized kitchen, a short repeatable menu, several shifts. The cook no longer improvises daily; he works from a written menu, even if the recipes live in the head of the boss. Level 2 is the restaurant with a set lunch menu: written procedures, fixed suppliers, stock control, an incident book. If the head cook falls ill, the second can run the service without carnage. Level 3 is the restaurant with an à la carte menu and formal service: traceability of every ingredient, allergens on the menu, health certification, contracts with suppliers, mandatory staff training. Level 4 is the kitchen specialised in a complex product (sushi, haute cuisine, artisan bakery): specific tools the normal restaurant does not need (wood-fired oven, special knives, proving chamber), calibrated processes, quality metrics. Level 5 is the restaurant with a Michelin star: the whole system works, the dish is the result of the organisation, not of one person’s talent.

The analogy holds to the end, including the most interesting detail: you can operate at any level, but the promises you can keep are different. The street stall cannot promise a consistent experience to 80 diners with a booking. The cluster at level 1 cannot promise productive multi-tenant LLM service with an SLA. In both cases the problem is not the technical capability of the last component (the griddle cooks; the pod starts); it is the organisational capability of the whole system.

Let us go level by level.

Level 0 — Chaotic: the server with Linux and nothing else

The capability it gives. Running containers with docker/podman, running binaries, connecting the server to the network. The operator can SSH in, do things, and see results.

The test for the level. “If I reinstall the server from scratch, can I leave it identical to how it was in one afternoon, using only saved notes?”. If the answer is no (because the steps are in the head of whoever built it, in .bash_history, in an out-of-date wiki), you are at level 0.

Minimum pieces to settle before moving up to level 1.

Piece	Suggested decision in 2026	Why it matters when moving up
Linux distribution	Stable Debian or Ubuntu LTS	Long, predictable support, recent kernel available
Kernel	Recent LTS (≥ 6.6) with BPF and modern schedulers	Cilium/eBPF and recent NVIDIA drivers demand it
NVIDIA drivers	The version that matches the CUDA of the LLM engine you will serve	A driver/CUDA mismatch blocks vLLM before you start
Container runtime	`containerd`	CNCF standard, integrated with RKE2/kubeadm
Root filesystem	XFS or ext4 + LVM thin pools	Snapshots, online expansion
Time synchronisation	`chrony` with your own servers	TLS, correlated logs and short certificates demand it
Management network	Dedicated VLAN, ACLs on the switch	Isolate the control plane from workload traffic
Cluster network	LACP + jumbo frames + BGP (if you are going with Cilium)	Intra-node NVLink does not rescue the service network
BMC / IPMI	Out-of-band access with TLS and MFA	Recovery when the operating system does not boot

Antipatterns that pin you at level 0.

Pet servers (with their own names, configured by hand, not replaceable).
Changes applied with vi directly over /etc/... with no commit to a repo.
Deployment with docker-compose with no healthchecks and no automatic restart.
An inventory that lives in an Excel sheet nobody updates.

Deployment order within the level. System image from PXE/cloud-init with the initial configuration (LVM, hostname, network, SSH key, chrony) → bastion/jump host bootstrap → inventory in Ansible (or an equivalent declarative tool, even if it is later replaced) → NVIDIA drivers + container runtime → smoke test (a CUDA container passes nvidia-smi). At this point, the server is ready for Kubernetes to come in.

Level 1 — Repeatable: an operational Kubernetes cluster

The capability it gives. Scheduling containers with a scheduler, network abstraction between pods, persistent volumes, workload lifecycle, manual horizontal scaling.

The test for the level. “Can I lose a node and have the workloads rescheduled without human intervention?”. If yes, you are at level 1. If not, because the pods are pinned to nodes, because there are no replicas, because the PVCs do not reattach, you are still at 0 with Kubernetes on top.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Main alternative
k8s distribution	RKE2 (CIS-hardened by default, no commercial surcharge)	k3s for very small edge, plain kubeadm for custom cases
CNI	Cilium with kube-proxy replacement, BGP, Gateway API	Calico (without BGP it does not compete with Cilium in 2026)
CSI block + filesystem + object	Rook-Ceph (RBD + CephFS + S3-compatible RGW)	OpenEBS Mayastor + Garage for small deployments
Ingress	Cilium Gateway API (better to unify with the CNI)	NGINX Ingress, Traefik
Basic certs	Self-signed bootstrap	(cert-manager comes in at level 3)
Workload handling	`kubectl apply` + Helm from the terminal	No GitOps yet
Container registry	Any internal registry (or a trusted external one) with TLS	(a managed internal registry comes in at level 2)

Antipatterns that drop you to level 0.

Services deployed with kubectl apply from one person’s terminal and with the YAML saved nowhere.
Persistent volumes with no backup policy.
A “single-node cluster” as permanent production, an architectural single point of failure.
A CNI with no NetworkPolicy available or no BGP when the network requires it.

Deployment order within the level. RKE2 installed on at least three nodes for an HA control plane → Cilium installed in kube-proxy replacement mode + BGP control plane → Rook-Ceph on at least three nodes covering block (RBD) + filesystem (CephFS) + object (S3-compatible RGW) with 3× replication or Erasure Coding depending on the pool → smoke test (a Deployment with a PVC starts, the pods reschedule when a node is cordoned, the data persists).

Level 2 — Defined: the cluster rebuilds from git

The capability it gives. The cluster state lives in a repository. Every change goes through a commit. Anyone can rebuild the cluster (or an equivalent one) from the repo and the backups. Basic observability warns when something breaks.

The test for the level. “If I lose the whole cluster, can I recreate it in X hours from the repo + the backups, with no manual intervention beyond the bootstrap?”. The hours are negotiable; what defines the level is that the repo + the backups are enough, not that the person-who-knows is available.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
Forge	Forgejo (or Gitea, GitLab CE)	Self-hosted OSS, community fork of Gitea, open governance
GitOps reconciler	Flux	CNCF graduated, native multi-tenancy, lightweight
Image registry	Forgejo Container Registry	Next to the code, no extra piece
Metrics TSDB	VictoriaMetrics + vmagent	Higher throughput than plain Prometheus, long retention, PromQL compatible
Visualisation	Grafana	The de facto standard
Logs	Loki or Vector	OSS, integrated with Grafana
Alerting	Alertmanager + Keep (OSS orchestrator)	Keep adds multi-channel routing with no lock-in
DB backups	Barman Cloud (Postgres)	The standard for CNPG
Object / dataset backups	Ceph RGW multisite + CephFS snapshots	Cross-pool and cross-site

Antipatterns that drop you to level 1.

kubectl apply run in production outside the repo (undetected drift).
main branches with write permissions for humans without review.
A monolithic repo with no tenant/infra/apps separation (cross changes that cannot be audited).
Metrics kept no longer than 7 days (no SLO observable over a month).
Alerting that fires for everything (fatigue) or for nothing (silence).

Deployment order within the level. Forgejo deployed first (it is a prerequisite for everything else) → Flux installed and pointing at the manifests repo → an initial repository with the Cilium and Rook-Ceph Helm releases reconciled by Flux (replacing the level 1 kubectl apply) → VictoriaMetrics + Grafana + Loki via Helm/Flux → Postgres backups and Ceph snapshots scheduled → smoke test (tear the cluster down, restore from repo + backup, the services come back).

Level 3 — Managed: identities, certificates, secrets and policies

The capability it gives. Every human operating the cluster does so with their own identity (no shared kubeconfig), with MFA and with limited permissions. Automatic internal TLS. Encrypted versioned secrets. Policies that reject insecure configurations before they reach the cluster. A complete audit of who did what.

The test for the level. “If an attacker gets hold of an administrator’s laptop, what can they do in production?”. At level 3 the answer is “little”: MFA blocks the second factor, Kyverno policies block destructive changes without approval, the NetworkPolicies prevent lateral movement, the secrets are encrypted with an external KMS, and the audit log remains. At level 2, “everything”.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
IdP / OIDC	Defguard	Spanish OSS, WireGuard + OIDC + 2FA, multi-org
Federation with the cluster	OIDC in kube-apiserver, OIDC in Forgejo, OIDC in Grafana	Consistent SSO
Internal PKI	cert-manager + Trust Manager	De facto standard, ACME and internal CA
External ACME	Let’s Encrypt for edge certs	No cost, automated
Secrets in git	SOPS + age or an external KMS	Versionable, encrypted in the repo
Secret syncing	External Secrets Operator	Pull from KMS / Vault into the cluster
Policy as code	Kyverno (or OPA Gatekeeper)	Kyverno has a shallower learning curve
NetworkPolicy	Cilium NetworkPolicy + L7	Default deny per namespace
Runtime security	Tetragon (Cilium)	eBPF, complements NetworkPolicy with detection
Vulnerability scanning	Trivy in the CI pipeline + admission	SBOM per image, blocking of critical CVEs
Audit log	kube-apiserver with `--audit-policy-file` sent to Loki	Regulatory traceability

Minimum Kyverno policies to keep alive.

Deny :latest images or images without a sha digest.
Deny pods without securityContext.runAsNonRoot=true.
Deny pods without resources.limits (CPU + memory).
Deny Services without an owner=<team> label.
Deny changes in critical namespaces (kube-system, flux-system) without an approval label.

Antipatterns that drop you to level 2.

A kubeconfig shared between administrators.
Secrets in plain data: in a manifest committed to the repo.
NetworkPolicy absent from new namespaces by default (implicit allow-all).
kubectl edit or kubectl patch in production without going through the repo.

Deployment order within the level. Defguard deployed and enrolled with WireGuard / OIDC → OIDC integration with kube-apiserver, Forgejo and Grafana → cert-manager installed and issuing internal certificates (your own CA for mTLS, Let’s Encrypt for the edge) → SOPS configured and External Secrets Operator installed → migration of secrets from plain to encrypted → Kyverno with initial policies in audit mode, then enforce → default-deny NetworkPolicy per namespace → Tetragon enabled → smoke test (try to bypass each policy and check that admission rejects it).

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

The capability it gives. The Kubernetes scheduler sees the GPUs, distinguishes them, can partition them (MIG) or multiplex them (time-slicing), expose DCGM metrics, autoscale with KEDA using metrics from the LLM workload itself (vllm:num_requests_running, vllm:gpu_cache_usage_perc), and carry traces with the GenAI semantic conventions. Everything needed for the LLM inference stack to rest on a platform that understands its nature.

The test for the level. “If I deploy a pod asking for nvidia.com/gpu: 1, does it get scheduled on the right GPU, with the right slice, with DCGM metrics exposed, with GenAI observability ready to receive spans?”. If yes, you are at level 4. If the answer requires “it depends which node and who deploys it”, not yet.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
GPU device plugin	NVIDIA GPU Operator	Deploys drivers, container toolkit, DCGM and MIG manager with one operator
HW partitioning	MIG (Multi-Instance GPU) on the H100 where it applies	Real hardware isolation, not time-slicing
GPU metrics	DCGM Exporter	SM utilisation, VRAM, temperature, throttling, NVLink bandwidth
LLM metrics	vLLM Prometheus endpoint + scrape	TTFT, TPOT, KV cache, prefix hit rate
Autoscaling	KEDA with a Prometheus ScaledObject	Scales on LLM metrics, not on CPU
LLM operators	vLLM Production Stack / OME (Operator Model Engine)	Declarative handling of models / adapters
Traces	OpenTelemetry Collector with OTLP receivers + processors + exporters	`gen_ai.*` semantic conventions (post)
LeaderWorkerSet	LeaderWorkerSet API (k8s 1.30+)	Tensor parallel topology coherent with NVLink
Topology Manager	enabled with `single-numa-node`	Pinning GPU pods to the right NUMA node

The key decision: MIG, time-slicing or passthrough.

MIG divides an H100 into 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb (slices with real HW isolation). Useful for serving several small models or reserving capacity per tenant with a guarantee. Limitation: up to 7 instances per GPU, predefined profiles.
Time-slicing shares a GPU between several pods with no HW isolation. Useful for dev/test, not for multi-tenant production with an SLA.
Passthrough assigns the whole GPU to one pod. Useful for tensor parallel across several GPUs on the same node (a large LLM with TP=4).

For a productive LLM platform, the practical rule: passthrough for the large models with TP, MIG for embeddings and small models that cohabit, never time-slicing in production.

Antipatterns that drop you to level 3.

Installing NVIDIA drivers by hand outside the GPU Operator (silent breakage when Kubernetes is upgraded).
Serving an LLM with requests.gpu: 1 without having decided on MIG / passthrough (you end up with idle GPUs from fragmentation or pods stepping on each other).
KEDA autoscaling on CPU (the classic HorizontalPodAutoscaler) on pods that are almost always at 10 % CPU but 95 % KV cache.
OpenTelemetry deployed but without the gen_ai.* semantic conventions (the traces are not LLM-aware).

Deployment order within the level. NVIDIA GPU Operator installed via Helm/Flux with the driver version that matches the chosen LLM engine → DCGM Exporter enabled and metrics visible in Grafana (NVIDIA dashboards imported) → MIG manager configured for the nodes where it makes sense (a typical mix on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs + embeddings) → OpenTelemetry Collector with attributes processors to enrich spans with your own labels (tenant_id, priority_tier) + exporters to Langfuse and to Tempo → KEDA installed with an example ScaledObject pointing at vllm:num_requests_running → vLLM Production Stack or OME to declare models as a CRD → smoke test (a vLLM Deployment declared via CRD starts, serves a token, exposes metrics, the trace reaches Langfuse, KEDA scales under synthetic load).

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

Once at level 4, the cluster meets the contract that the post on the seven layers assumed as its starting point. Level 5 adds no infrastructure: it adds the LLM stack proper. For completeness, the seven components of level 5 are:

Gateway (Envoy AI Gateway) — it comes first, directing traffic to LLM inference and embeddings.
LLM inference (vLLM Production Stack or OME with vLLM) — on the GPUs already discovered by the level 4 GPU Operator.
Embeddings + reranker (Infinity, TEI) — a pod separate from the LLM, already covered in the previous post.
Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets) — most of it already existed at level 2 as data; now it specialises for RAG.
LLM-aware observability (Langfuse) — it plugs into the level 4 OTel chain.
GitOps control plane — the one from level 2 remains the only legitimate authority.
Dependency tracking (Hubble flows + Otterize) — on the Cilium that already existed at level 1.

The criterion for promoting from level 4 to level 5 is not technical: it is contractual. The cluster already supports LLMs; the decision is when to open real customer traffic. Promotion demands: a green golden eval of the model, a signed incident runbook, negotiated SLOs, a continuity plan, and mapping to ENS / NIS2 / 42001 where it applies.

The maths that matter: what skipping a level costs

To quantify the thesis of the post, an order-of-magnitude estimate of the cost of moving up each level in time versus moving up after production is running. The figures are engineering time with a small platform team (2-3 people), assuming templates and prior experience.

Level	Time to build on an empty cluster	Time to retrofit with production running
0 → 1	1-2 weeks	1-2 weeks (little downstream refactoring)
1 → 2	2-3 weeks	4-8 weeks (migrating everything to git)
2 → 3	2-4 weeks	8-16 weeks (image rebuilds, secret migration, retroactive RBAC)
3 → 4	1-2 weeks	4-8 weeks (reconfiguring GPUs, moving models to MIG, instrumenting `gen_ai.*`)
4 → 5	1-2 weeks	2-4 weeks
Total 0 → 5	~10-15 weeks	~20-40 weeks if done in the wrong order

The typical multiplier observable in practice: 2× to 3× the cost if it is done in the wrong order. And that assumes it gets done at all, since many projects never get past level 2 because “the identity thing” can always wait for another sprint. By the time the incident arrives, it is too late to start.

Beyond time, the operational cost (night-time incidents, security escapes, invisible debt) grows exponentially with the gap between the actual level and the required level. A cluster at level 2 serving productive LLMs to regulated customers is a time bomb: technically it works, organisationally it does not.

Final diagram: the complete ladder with pieces

The ladder is not decorative: each level enables the next. You cannot have LLM-aware observability (level 4) without OTel deployed via Flux (level 2). You cannot have automatic internal TLS (level 3) without a root PKI that lives somewhere (registry and certificates managed from level 2). You cannot have KEDA scaling on vLLM metrics (level 4) without Prometheus / VictoriaMetrics scraping (level 2). The levels are not a conceptual hierarchy: they are a hierarchy of installation dependencies.

Typical design decisions that break progress

Mistakes seen repeatedly that drag the cluster back a level:

1. Jumping from level 1 straight to level 4. “We are in a hurry to serve the LLM, we will do the identity and GitOps stuff later.” Later is always two orders of magnitude more expensive and always arrives after the first incident.

2. Confusing Helm with GitOps. Having Helm charts is not level 2. It is level 1 with templates. Level 2 demands that a reconciler (Flux/ArgoCD) applies the charts from a repo, detects drift and warns.

3. cert-manager with no usage policy. Having auto-renewed certificates but using TLS only at the ingress, with no internal mTLS between services, leaves the TLS promise lame and turns level 3 into a cosplay of level 3.

4. NVIDIA drivers by hand. It works on day one and breaks the day of the first kernel upgrade. The rule: drivers always via the GPU Operator, never operating system packages.

5. Prometheus metrics but 7 days of retention. Without long retention (≥ 90 days) there is no honest SLO. VictoriaMetrics with a year of retention costs little more than Prometheus with 7 days, and it unlocks compliance and serious postmortems.

6. OIDC only for kube-apiserver. If Forgejo, Grafana, Defguard and vLLM each have their own auth system, you do not have SSO, you have islands. An honest level 3 demands federation.

7. Kyverno in permanent audit mode. Policies that do not reject are not policies, they are alerts. At some point you have to move to enforce. Until then, you are still at level 2 wearing a level 3 face.

8. MIG with no conscious decision on the profile. Configuring MIG with the default profile without having measured the size of the models that will cohabit leaves fragmented GPUs with slices nobody uses. The rule: MIG only if you have measured and decided the profiles in advance.

They all share one root: declaring the level without passing the level’s test. Saying “we already did GitOps” when things are still applied with kubectl edit in prod. Saying “we already did identity” when there is a shared admin kubeconfig. Saying “we are ready for LLMs” when there is no DCGM Exporter and no Langfuse plugged in.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), a reasonable setup after passing the five levels distributes the components like this:

control plane (3 nodes without GPU, hostnames cp-01..03)
├── kube-apiserver, etcd, controller-manager, scheduler
├── Flux, Forgejo, cert-manager, External Secrets, Kyverno
└── Tetragon (DaemonSet here too)

worker plane (≥ 3 nodes without GPU, hostnames worker-cpu-01..03)
├── Cilium agent (DaemonSet)
├── Rook-Ceph OSDs + MONs + MDS (CephFS) + RGW (S3)
├── VictoriaMetrics + Grafana + Loki
├── Defguard (StatefulSet)
└── Langfuse + OTel Collector

GPU worker plane (≥ 2 nodes with 4×H100 SXM, hostnames worker-gpu-01..02)
├── NVIDIA GPU Operator (driver + container toolkit)
├── DCGM Exporter (DaemonSet)
├── MIG manager (configuring the chosen profile)
├── vLLM (Deployment) — general LLM TP=4 occupies 4 GPUs (passthrough)
├── vLLM (Deployment) — code LLM TP=2 occupies 2 GPUs
├── Infinity (embeddings) — 2 replicas cohabit on 2 MIG slices
└── KEDA scaler listening to vLLM metrics

The operational rule: the control plane and the CPU plane are separated from the GPU plane. An incident on the GPU plane must not take the control plane with it (the control plane is what recovers the cluster). And the CPU plane concentrates everything that moves relevant state (Forgejo, Rook-Ceph, Postgres CNPG, Langfuse): it is the heart to protect.

The GPU hardware specialises as much as possible: GPU pods run only on GPU nodes, and GPU nodes run nothing CPU-bound beyond the operational overhead (Cilium, GPU Operator, DCGM). This is enforced with nodeSelector + taints/tolerations + a Kyverno policy that rejects pods with no GPU requests being scheduled onto GPU nodes.

What we have not covered (upcoming posts)

This post walks the vertical path upwards. Some horizontal and cross-cutting pieces still deserve an article of their own:

Multi-site active/standby: how two clusters are federated with Cilium Cluster Mesh and what changes at each level when there are two sites instead of one.
Migration between levels with real traffic: how a cluster already in production is retrofitted to the next level with no downtime.
Day-to-day operation: runbooks per level, which dashboards to look at each morning, which SLOs to define per component.
The cost plane: what each level costs in hardware, energy, engineering hours, optional OSS licences (commercial support from Rancher, Cilium Enterprise, etc.) and when each expense is justified.
Operationalised compliance: how levels 3 and 4 map to ENS High, NIS2 and ISO/IEC 42001 controls without turning the cluster into a paperwork exercise.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook-Ceph — rook.io
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
vLLM Production Stack — docs.vllm.ai/en/latest/serving/production_stack.html
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
CIS Kubernetes Benchmark
NIST SP 800-207 — Zero Trust Architecture

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

Sat, 30 May 2026 16:00:00 +0200

TL;DR

The on-premise LLM inference stack is not a model server: it is a building of seven layers that hold each other up. The inference layer (vLLM / SGLang) serves tokens, but without a dedicated embeddings layer RAG does not work; without the gateway layer the client couples its SDK to the specific engine; without an LLM-aware observability layer (Langfuse + OpenTelemetry GenAI) any quality degradation goes unnoticed; without a GitOps control plane layer (Flux + Forgejo) any manual change leaves invisible debt; and without a dependency tracking layer (Hubble + intent-based policies) decommissioning a Service silently breaks applications nobody remembers were using it. This post is born from a concrete incident: a pipeline that reported status: completed and matched_jobs: 0 for days because it kept calling an Ollama that had already been scaled to zero, while a badly written except labelled the generic ConnectError as “ChromaDB indexing error” and the vector store, innocent, took the blame. Each of the three symptoms was the cry of a layer that was missing or badly designed. The body of the post walks the seven layers with their canonical OSS piece, the design decisions that break them, the sizing maths on a generic 4×H100 SXM cluster (320 GB of VRAM, NVLink) and a final diagram of the connected stack. The thesis: a stack that passes the incident test is not measured by its peak throughput, it is measured by how long it takes to shout when something drifts from the design.

You are here: the seven layers seen from above

Before going into detail, the map. The seven layers are not seven servers: they are seven responsibilities the stack has to cover. A layer can collapse into one pod (gateway) or spread across several components (observability = traces + metrics + flow logs). What it cannot do is be missing.

Layers 1–4 sit on the request path: if they fall, the client notices in seconds. Layers 5–7 sit on the path of design and operations: if they fall, there is no immediately visible error, and that is why they are the ones that produce silent incidents, which is how most serious incidents end up. This post argues that the quality of the stack is measured in layers 5, 6 and 7, because 1–4 are commodities where everyone picks roughly the same pieces.

The analogy: the office building with shared services

Picture a twelve-storey office building with several tenants. It has a concierge desk (gateway: filters who comes in), it has lifts (LLM inference: they move the heavy load), it has stairs and a goods lift (embeddings: light but constant traffic, hardly anyone notices them until they break), it has plumbing and water tanks (vector store: what actually holds the state), it has a distribution board and sensors (observability: what warns you when something is drawing more than planned), it has a building manager (GitOps control plane: the only legitimate authority to move anything), and it has a tenant register (dependency tracking: who is connected to which shared service).

When a new client moves into the third floor and asks to install a server that needs more amperage than planned, the problem is not electrical: it is administrative. If the client can go straight to the board and plug in whatever they like, the building survives for a while and then a breaker trips at three in the morning. If the client has to go through the manager, the manager consults the tenant register (is anyone else hanging off that same circuit?), reviews the electrical planning (are we at the limit?) and either authorises or redirects. The building stays standing not because of its electrical installation but because of the discipline of going through the manager.

The LLM inference stack works identically. The physical layers (1–4) are the visible ones, the ones marketing people put on the slide. The governance layers (5–7) are the ones that distinguish a platform from a pile of pods with good luck.

Now to the incident that motivates the whole post.

The hook: the log that lied for six days

The application is called jobhunter internally. It is a cron pipeline that every six hours sweeps public job-ad sources, filters by EU geography, embeds the new ads, indexes them in a vector store and matches them against search profiles. The last step fires notifications.

For six days the pipeline reported the same thing on every run:

status: completed
total_found: 756
new: 23
matched_jobs: 0 ← zero, run after run

And in the logs:

[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/delete → 200 OK
[INFO] matcher: Purged 12 expired jobs from ChromaDB
[ERROR] pipeline: ChromaDB indexing error: All connection attempts failed
[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/get → 200 OK
[ERROR] pipeline: Matching error: All connection attempts failed

This is a log that invites you to blame ChromaDB. And in fact the first post-mortem written internally pointed at a version incompatibility between the Python client and the v2 server. A reasonable hypothesis, technically plausible, completely false.

The real cause: weeks earlier the general LLM on the platform had been migrated from Ollama to vLLM. The migration was clean for the two applications that depended directly on the large model, their manifests were pointed at the new endpoint. What nobody did was look at who else was calling the Service ollama.ollama.svc:11434. jobhunter called it to generate the embeddings for the ads. When the Ollama deployment was scaled to zero, the Service was left empty, and any outbound connection received a generic ConnectError("All connection attempts failed") from httpcore. The try/except wrapping the whole matching stage caught the exception and labelled it “ChromaDB indexing error”, because that was the lexical wrapper of the block, not because ChromaDB had anything to do with it. ChromaDB was answering 200 to delete and get in those same logs.

Three factors kept the incident alive for six days:

The except named the wrapper rather than the stage that failed. The log said “ChromaDB indexing error” when the error was an embeddings call against a nonexistent Service.
The pipeline returned status: completed even when there were errors. No alerting based on status fired. The metric that would have fired (matched_jobs stuck at zero) was not instrumented.
The pipeline image ran as :latest with no versioning, no SBOM, no reproducibility. When it started failing, it was impossible to know which Ollama client it had been built with.

Each of the three symptoms is the cry of a layer that was missing. The first asks for LLM-aware observability that distinguishes stages (layer 5). The second asks for the pipeline logic and the Prometheus metrics to behave like contracts (layer 5, SLI/SLO dimension). The third asks for GitOps with pinned images and an auditable SBOM (layer 6). And the whole incident, decommissioning a Service without knowing who consumes it, shouts dependency tracking (layer 7).

The rest of the post walks the seven layers through that lens: what each one solves, which OSS piece implements it in 2026, and which typical design decision breaks it.

Layer 1 — Gateway: the client SDK must not couple to the engine

What it has to solve. That the client sends POST /v1/chat/completions with the standard OpenAI SDK and never learns which engine (vLLM, SGLang, TensorRT-LLM), which specific model, which LoRA adapter or even which GPU pool is serving the request. Authentication, rate limit, routing by body.model and by tenant, header injection for tracing.

Canonical OSS piece in 2026. Envoy AI Gateway (Envoy with GenAI extensions: routing by model, token-based rate limit, fallback chains) or Cilium Gateway API with your own filters. For cases where the client wants multi-provider without distinguishing on-prem from SaaS, LiteLLM Proxy is the lightweight equivalent.

What breaks the layer. Exposing the inference engine endpoint directly. If clients call http://vllm-prod.svc:8000, any change of engine, model or pool forces you to touch the code of every app. The rule: the engine changes, the contract does not. The OpenAI SDK is standard; the routing behind the gateway is where design freedom lives.

Where jobhunter failed. There was no gateway. The app called ollama.ollama.svc:11434 directly. When Ollama died, there was no intermediate layer that could answer with a fallback, a retry against another pool, or at least a 503 error with a descriptive body.

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

What it has to solve. Serving tokens with throughput and latency under control: continuous batching, PagedAttention so the KV cache does not fragment the VRAM, FP8 so a 32B model fits with room to spare, multi-LoRA for per-tenant customisation without replicating the base, structured output for function calling with schema guarantees.

Canonical OSS piece in 2026. vLLM is the default choice: it covers the state of the art (PagedAttention, continuous batching, FlashAttention, FP8 quantisation, multi-LoRA, structured output, speculative decoding). SGLang comes in when the workload has high prefix caching (long chat with large system prompts, agents with repeated instructions), because its RadixAttention composes better than standard prefix caching. For very specialised inference with proprietary NVIDIA kernels, TensorRT-LLM, accepting the hardware lock-in.

What breaks the layer. Assuming a single model “does everything”. A 32B chat LLM does not serve /v1/embeddings; if you try, vLLM answers BadRequestError: "The model does not support Embeddings API". Assuming it did was one of the concrete wounds of the incident: the app expected an endpoint the model did not implement.

The design decision that costs more later. Serving the LLM with aggressive quantisation (INT4) without a quality eval calibrated for your corpus. INT4 with AWQ or GPTQ saves VRAM, but it degrades answers in technical or legal Spanish in a measurable way. The rule: any quantisation change goes through the same golden eval as a model change.

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

What it has to solve. Generating dense vectors for the RAG corpus and for retrieval queries. The embeddings model is a different thing from the chat LLM: different architecture (encoder, not decoder), different size (hundreds of millions of parameters, not tens of thousands of millions), different API (an /embeddings endpoint that takes text and returns a fixed-dimension vector).

Canonical OSS piece in 2026. Infinity or Hugging Face Text Embeddings Inference (TEI) to serve models of the bge-*, multilingual-e5-*, nomic-embed-* families with high throughput and multi-model support. OpenVINO Model Server when Intel hardware is available. sentence-transformers as a fallback embedded in the application itself when the corpus is small and the deployment is constrained.

What breaks the layer. Treating it as “the LLM does that too”. It does not if it is chat-only; and even when it does (some models have a dual endpoint), mixing chat and embeddings serving in the same process punishes the throughput of both. The physical separation is the design.

The technical fact people forget. The vector store and the embeddings model form an indivisible unit. Changing the model from multilingual-e5-large (1024 dim) to multilingual-e5-small (384 dim) is not a substitution: it is creating a new collection (my_corpus_v2) and re-embedding the whole corpus. If you upsert into the old collection, you hit a dim mismatch at runtime that takes the pod down. This seems obvious and is violated constantly, because the embeddings model is chosen once and then forgotten.

Layer 4 — Vector store + relational data + storage: what actually holds the state

What it has to solve. Persisting the vectors with efficient filters (tenant_id, created_at, source), persisting the relational metadata (users, configs, versioned prompts, traces), and persisting the model weights, the LoRA adapters, the datasets and the original corpora.

Canonical OSS pieces in 2026.

Qdrant for large collections (>200 k vectors), payload-aware filters, multi-tenancy via collection or via field.
pgvector for small collections with mandatory relational joins (wanting to do WHERE doc.author = ... AND vector <=> $1 in the same SQL).
PostgreSQL operated by CloudNativePG (CNPG) for the relational side: Barman Cloud backups, replication, connection via pooler.
MinIO for S3-compatible objects: bucket per tenant, cross-site replication, weights and adapters versioned by sha256.
Redis for queues, rate-limit counters and a cache of frequent retrievals.

What breaks the layer. Assuming the vector store is stateless ephemeral. It is exactly the opposite: it is the component where losing state costs the most. Without verified backups of the vector store, an index corruption forces you to re-embed the whole corpus, and on a corpus of millions of documents that is hours or days of GPU.

The design decision that pays later. Forgetting to version the collection by dimension and embeddings model. Suggested convention: my_corpus__embed-multie5l__1024d__v3. The name carries metadata; any change in any of the three attributes forces a new collection. It is ugly but it protects against an accidental upsert with the wrong dim.

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

What it has to solve. That any inference can be recovered from its trace_id, with all the gen_ai.* attributes (semantic conventions) plus your own attributes (tenant_id, adapter_id, priority_tier), a breakdown of latency (queue → prefill → decode → network), tokens consumed, tools invoked, exact model and adapter. And in parallel: Prometheus metrics from vLLM (vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:prefix_cache_hit_rate), from the GPU (DCGM Exporter), from the network (Hubble flow logs with drops and NetworkPolicy enforcement).

Canonical OSS pieces in 2026.

OpenTelemetry Collector as the single transport for traces, metrics and logs, with OTLP receivers and separate exporters per destination.
Langfuse self-hosted for the LLM-aware side: tracing, prompt versioning, evals with LLM-as-judge (post and post).
VictoriaMetrics + Grafana for high-throughput TSDB metrics with long retention.
Hubble (Cilium) for L3/L4/L7 flow logs and NetworkPolicy visualisation.
DCGM Exporter for GPU metrics.

What breaks the layer. Wrapping the whole pipeline stage in a single try/except that labels the exception with the wrapper’s name. The operational rule: one try/except per stage, with the stage label in the message and a Prometheus metric with labels={"stage": "<name>"}. That way “ChromaDB indexing error” would never have been the log for an embeddings failure; it would have been “Embeddings call failed: ConnectError(ollama:11434)”.

Complementary rule. The pipeline returns status: completed if and only if there were no errors. With errors it returns completed_with_errors or failed, and the metric pipeline_errors_total{stage} is incremented. An alert based on increase(pipeline_errors_total[1h]) > 0 fires before the second failed run. Without this discipline, observability exists but does not warn.

Layer 6 — GitOps control plane: the only legitimate authority

What it has to solve. That the cluster state is the state declared in git. That any divergence between git and the cluster is visible and, in critical components, auto-reconciled or auto-alerted. That every deployed image has an immutable tag (sha digest or semver pin), an SBOM (Trivy) and traceability back to the commit that introduced it.

Canonical OSS pieces in 2026.

Flux (or ArgoCD) as the reconciler.
Forgejo (or Gitea, GitLab CE) as the self-hosted OSS forge.
cert-manager + Trust Manager for internal PKI.
External Secrets Operator + SOPS for encrypted versioned secrets.
Kyverno (or OPA Gatekeeper) for binding policies: deny :latest images, deny pods without a NetworkPolicy, deny Services without an owner label.
Trivy for SBOM and vulnerability scanning of images in the CI pipeline.

What breaks the layer. Images with a mutable tag (:latest, :main). Any kubectl edit in production that is not reflected in git. main branches with write permissions for humans without review. The rule: if a human can mutate the cluster without going through a signed commit, you do not have GitOps, you have somebody’s whiteboard.

How it applies to the incident. If the pipeline image had been pinned to a sha digest (registry.interno.local/jobhunter@sha256:9af2...), the team could have audited immediately which Ollama client it carried. With :latest, not even that.

Layer 7 — Dependency tracking: the layer the incident exposed

What it has to solve. Knowing who calls which Service, both declaratively (what the gitops repo says) and observed (what has been seen crossing the network over the last N days). And, on mature platforms, propagating that information as policy: if nobody declares and nobody observes traffic to the Service ollama.ollama.svc, decommissioning it is safe; if somebody declares it or uses it, decommissioning it opens a ticket.

Canonical OSS pieces in 2026.

Hubble (Cilium) for observed flow logs: hubble observe --to-namespace ollama --since 14d gives the list of source namespaces that have talked to Ollama in the last two weeks.
Otterize for intent-based policy: each Deployment declares “I need to talk to ollama-svc”, and the operator generates the corresponding NetworkPolicy and maintains a browsable catalogue of who is trying to talk to what.
manual kubectl-grep as a fallback: kubectl get deployments,cronjobs,statefulsets -A -o yaml | grep -E 'ollama[.-]' produces the declarative list.
NetworkPolicy as code reviewed in CI: every PR that touches a Service requires the associated policy to be kept or explicitly updated.

A pre-decom checklist that the incident suggests codifying as a CI hook:

SVC="ollama.ollama.svc.cluster.local"

# (a) declarative grep in the gitops repo
git -C $GITOPS_REPO grep -l "$SVC" || echo "OK declarative"

# (b) observed grep in Hubble (last 14 days)
hubble observe --to-fqdn "$SVC" --since 336h --output json \
 | jq -r '.source.namespace' | sort -u || echo "OK observed"

# (c) live grep in the cluster
kubectl get all -A -o yaml | grep -E "$SVC" || echo "OK live"

If all three return empty, the decom is safe. If any of them has content, there is downstream debt still open. The jobhunter incident is exactly what happens when this check does not exist: the team that decommissioned Ollama looked at the list of applications it knew depended on it directly; nobody looked at the list of those that depended on it silently.

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

The generic reference cluster for everything that follows: 4×H100 SXM 80 GB, NVLink between the four, 640 GB of system RAM, 2×NVMe NVMe-oF for local storage of weights and caches, redundant 25/100 GbE to the switch. Aggregate VRAM: 320 GB.

The VRAM budget is not free. A first reasonable split for a stack that serves one large general LLM, a medium code-specialised model, embeddings and a reranker, with room for multi-LoRA and for the KV cache:

Component	Reference model	Quant	Model weights	Reserved KV cache	Total VRAM
General LLM (TP=4)	70B-instruct	FP8 W8A8	70 GB	60 GB	130 GB
Code LLM (TP=2)	32B-coder	FP8 W8A8	32 GB	28 GB	60 GB
Embeddings	multilingual-e5-large	FP16	1.3 GB	n/a	8 GB (×2 replicas)
Reranker	bge-reranker-v2-m3	FP16	0.6 GB	n/a	4 GB
Multi-LoRA pool (over the general LLM)	up to 16 adapters	bf16	16 × 0.4 GB ≈ 6 GB	reuses the LLM’s KV	6 GB
Reserved for fragmentation + overhead					~30 GB
Total committed					~238 GB / 320 GB
Free headroom					~82 GB

The 26 % of free headroom is not waste: it is what lets the vLLM scheduler avoid preempting requests under moderate pressure, lets continuous batching group large batches without aborting, and lets a failover from the other site promote a standby without OOM.

Expected throughput, with the 70B general LLM in FP8 and tensor parallel 4, on an H100 SXM with continuous batching active and prefix caching at 35–55 % (typical in multi-turn chat with shared system prompts):

$$ \text{aggregate tokens/second} \approx 1500 \text{ to } 2500 $$

for concurrency between 32 and 64 requests, with sub-second TTFT P95 on short prompts (<2k tokens) and TPOT P95 around 40–60 ms/token as perceived by the client. These numbers are reasonable orders of magnitude, not guarantees: real throughput depends on the prompt mix, on whether speculative decoding (EAGLE-3) is active and burnt-in, and on the network cost between gateway and inference pods.

Embeddings throughput on two replicas of multilingual-e5-large with dynamic batching:

$$ \text{embeddings/second} \approx 3000 \text{ to } 6000 \quad (\text{optimal batch} \sim 64) $$

Enough to reindex a corpus of 1 million documents in roughly an hour, assuming chunks of 512 tokens and two chunks per document on average. For corpora of tens of millions of documents, re-embedding is done by delta via CDC over the source (covered in RAG corpus curation), not by full sweep.

Retrieval latency on Qdrant with HNSW (M=16, ef_construct=200) over a collection of 5 million 1024-dim vectors filtered by tenant_id:

$$ \text{P95 latency retrieve top-50} \approx 8 \text{ to } 25 \text{ ms} $$

Below the cost of cross-encoder reranking (bge-reranker-v2-m3 over top-50 = roughly 30–60 ms more), and below any call to the LLM. The bottleneck in a well sized RAG pipeline is never the vector store: it is the LLM’s decoding.

Final diagram: the complete connected stack

The solid lines are the request path: client → gateway → inference engine → vector store/embeddings → response. The dashed blue lines are telemetry: each component emits OTel to the collector, which routes traces to Langfuse, metrics to VictoriaMetrics and logs to Loki. The dashed red lines are reconciliation: the GitOps control plane keeps any layer in its declared state and warns of divergence.

The diagram is not decorative: every arrow is a stable contract between two layers. If one layer changes (vLLM → SGLang, multilingual-e5 → bge-m3, Qdrant → pgvector), the arrows hold. That stability of contracts is the architectural property that lets a team migrate components without breaking downstream apps.

Typical design decisions that break the stack

A short list of mistakes seen repeatedly in stacks that looked well designed on paper:

1. Coupling the client SDK to the inference engine. Dropping the gateway because “vLLM already speaks OpenAI-compatible” works on day one and hurts the day you have to add a fallback, a canary or a second model.

2. Sharing the LLM and embeddings endpoint. A qwen2.5-32b-Instruct is chat-only; BadRequestError: "The model does not support Embeddings API" is the cry of a design that confused the two layers.

3. Reusing the vector store collection when changing the embeddings model. Different dimensions do not allow upsert. Versioning the collection by (model, dim, version) is ugly but it saves the day of the change.

4. A try/except wrapping an entire pipeline with the wrapper’s label. The log lies because the label is lexical, not causal. Every stage in its own try/except with its own label and its own metric.

5. status: completed with errors. The pipeline has to distinguish completed, completed_with_errors and failed, and the alerting has to fire on the last two. Without this, observability exists in theory and does not warn in practice.

6. Images with a mutable tag. :latest and :main are not tags, they are aliases. Without a sha digest there is no reproducibility and no auditable SBOM.

7. Decommissioning a Service without a pre-decom check. The three-grep check (declarative + observed + live) takes two minutes and costs six days of incident when it is skipped.

8. Default limits.memory on pods that load models. A sidecar loading sentence-transformers + torch + tokenizer needs 2–4 GB; with limits.memory: 1Gi you hit OOM on the first pod restart, and sometimes with no alert because the liveness probe answers through another route.

All of them are variants of the same principle: the stack does not fail in its most expensive layer (inference, where nobody underestimates the cost), it fails in the cheap, boring layers (gateway, observability, GitOps, dependency tracking) where it is tempting to save.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), the suggested split into pods:

gpu-node-01 (4×H100 SXM, intra-node NVLink)
├── vllm-llm-general (TP=4) ~130 GB VRAM (4 GPUs)
└── (shares GPUs with the multi-LoRA pool on the same deployment)

gpu-node-02 (4×H100 SXM, second node)
├── vllm-llm-code (TP=2) ~60 GB VRAM (2 GPUs)
├── infinity-embeddings (×2) ~16 GB VRAM (shared on 1 GPU with optional MIG)
├── tei-reranker ~4 GB VRAM (cohabitant)
└── failover reserve ~120 GB VRAM free for canary / standby

Not everything fits comfortably on a single node; two nodes with two H100 SXM each would be enough for the conservative setup, and the rest of the cluster (CPU-bound: gateway, vector store, observability, control plane) runs on GPU-free nodes.

The operational rule: inference concentrates, the rest of the stack distributes. Concentrating inference maximises the use of NVLink (cross-GPU tensor parallel without going through PCIe); distributing the rest prevents an event on the GPU node from taking the control plane down with it.

An even more conservative configuration, for SMEs with a single 4×H100 SXM node as a starting point, serves the general LLM (TP=4) and embeddings/reranker cohabiting with MIG (Multi-Instance GPU to partition an H100 into hardware-isolated slices). The code LLM is deferred to a second phase. It is viable and cost-aware; what is not viable is doing without layers 5, 6 and 7.

What we have not covered (upcoming posts)

This post focuses on the static design of the stack. Some pieces still deserve an article of their own:

The multi-site active/standby plane: Cilium Cluster Mesh, cross-cluster Qdrant replication, realistic RTO/RPO, when active-active pays off and when it does not.
The continuous fine-tuning plane: how the LoRA pipeline closes the loop from production feedback to a promoted adapter, in the spirit of the retrain post.
The safety/guardrails plane: where Llama Guard fits, Presidio for PII and XGrammar for guaranteed structured output, connected to layer 1, the gateway.
The cost plane: gen_ai.usage.* instrumentation at tenant and model level, tokens/euro dashboards, GPU elasticity decisions via KEDA.
The compliance plane: how the stack maps to ENS High, NIS2 and ISO/IEC 42001 without turning the deployment into a compliance exercise that paralyses delivery.
Below the engine: the mini-series that opens the stack’s basement — the interconnect (NVLink/NCCL), the host (NUMA, hugepages, CPU isolation) and the RKE2 resource managers that pin each pod to the right NUMA node.

Each one falls into a different series on the blog and will be covered with the same discipline: a concrete piece, a justified decision, the typical mistake seen in practice.

References

The six-stage LLMOps pipeline — the general framework this stack operates in.
The OSS catalogue for LLMOps in six stages — a card per tool in the stack.
Anatomy of an LLM request in production — the same architecture seen from the perspective of an individual request.
OSS vs hyperscalers in LLMOps — a comparison with the AWS/Azure/GCP stacks.
LLM tracing with OpenTelemetry GenAI — the backbone of layer 5.
Multi-LoRA serving — the piece that gives per-tenant customisation without replicating the base.
RAG corpus curation and Reranker and hybrid retrieval — everything that goes into layers 3-4 so that RAG does not degrade.
Quantisation for LLM inference — the reason for the FP8 W8A8 in the sizing.

Relevant official documentation

vLLM Production Stack — docs.vllm.ai
SGLang RadixAttention — github.com/sgl-project/sglang
Envoy AI Gateway — aigateway.envoyproxy.io
Langfuse self-hosted — langfuse.com/docs/self-hosting
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
Hubble flow observability — docs.cilium.io/en/stable/observability/hubble
Otterize intent-based access — docs.otterize.com
Flux GitOps toolkit — fluxcd.io

The six-stage LLMOps pipeline: global architecture and a deep dive into every component

Thu, 21 May 2026 06:30:00 +0200

TL;DR

The first two posts in the series established the LLMOps landscape and went down into the detail of the data pipeline with Kafka. This post does the intermediate zoom: it draws the complete map of the system, a global architecture of a modern LLMOps setup with all the pieces the field has settled on in 2026, and goes into depth on each of the six canonical stages of the pipeline: Data, Tune, Eval, Deploy, Observe, Retrain. For each stage we give the operational sub-tasks, the dominant tools, the design decisions that always show up, and the specific traps that are seen repeatedly in production. And, most important operationally: every stage carries a “you are here” mini-map over the cycle, which will be reused in any later post in the series to place the reader. The idea: that anyone reading a post about fine-tuning, about prompt versioning, about eval gates or about drift detection can look at the mini-map and know immediately which piece of the larger system they are thinking about that day.

This is the third post in the MLOps series specific to LLMs. Previous ones: Landscape 2026 and RAG over Kafka. Here we move from “the what” and “one piece” to the whole map, with detail per stage.

The global architecture: the master map

Before going down into each stage, let us fix the whole map. What follows is the reference drawing of a production LLMOps system in 2026, with every component the field has settled on in its place:

What you see: the six large boxes are the stages; the solid arrows are the flow of the pipeline; the dashed arrow that goes from Retrain to Data is the feedback cycle that turns LLMOps into a living process rather than a project that ends. The grey band at the foot holds the cross-cutting components (observability, prompt versioning, MCP, gateway, schema) that run through every stage and connect to each one.

Three quick readings of the map:

Horizontal, top: the happy path, data → tune → eval. What happens while you prepare the model.
Horizontal, bottom: the service path, deploy → observe → retrain. What happens once the model is alive.
Vertical: the connection between the two floors. Eval gateway feeds Deploy; Observe feeds Retrain; Retrain returns to Data.

From here on, every stage will include a navigation mini-map (“you are here”) to place you in the complete cycle. Let us go through each one.

Stage 1 — Data: ingestion, transport, versioning, indexing

Operational sub-tasks

The Data stage is the most underrated and the one that blocks the most projects. Its sub-tasks:

Ingestion from heterogeneous sources: OLTP databases (Postgres, MySQL), external APIs, file shares, scraping, SaaS systems, application logs, internal messaging.
Change capture (CDC) in streaming if the data is dynamic. Debezium over Kafka, Flink CDC, modern alternatives such as RisingWave that reads the WAL directly.
Transformation (cleansing, dedup, normalisation, PII sanitisation).
Schema management: schema registry, compatible evolution, backward/forward compatibility.
Versioning of training datasets and golden datasets: DVC + lakeFS (unified in November 2025). Covered in detail in the dedicated data versioning post.
Indexing for RAG: chunking, embeddings, writing to vector stores. Covered in depth in the Kafka post.
Materialisation into analytical tables: Tableflow → Iceberg/Delta, for BI consumption and low-latency queries.

Dominant tools

Sub-task	Tools 2026
CDC	Debezium, Flink CDC, RisingWave
Transport	Kafka (Confluent Cloud, Redpanda, pure Apache)
Schema Registry	Confluent Schema Registry, Apicurio
Stream processing	Apache Flink, RisingWave, Kafka Streams
Data versioning	DVC + lakeFS
Vector stores	Milvus, Qdrant, Weaviate, pgvector, LanceDB
Materialised tables	Tableflow → Iceberg/Delta
Batch ETL/ELT (where it applies)	dbt + Snowflake/Databricks

Design decisions

The three decisions that always show up:

Batch vs streaming: the more dynamic the data, the more streaming. For static corpora (manuals that never change) a nightly batch is enough; for transactional data the agent needs to see minute by minute, streaming from day 1.

Embedding model: changing the embedding model invalidates every indexed vector. An architectural decision: pin the model and have an explicit migration plan (the dual-index pattern seen in the Kafka post).

Vector store: pgvector if you already run Postgres and you are below 10M vectors; Qdrant if you want mid-scale simplicity; Milvus if you need billions; Weaviate if you value native hybrid search.

Traps

Hardcoding connections to the source (with no abstraction): when the database changes (version, host, schema), you break the whole pipeline. An adapter layer from day 1.
No schema registry: topics start breaking silently.
Full reindexing whenever something changes: it costs hours or days. Design the dual-index pattern from the start.
Unsanitised PII: the RAG is serving sensitive data without meaning to. Anonymisation in the pipeline, not at consumption time.

Stage 2 — Tune: preparing the model for your case

Operational sub-tasks

Base model selection: Llama, Qwen, Mistral, Gemma, DeepSeek according to licence, size, quality in your domain.
Dataset preparation: train/val/test split, format (chat templates, JSONL), augmentation where it applies.
Adapter configuration: LoRA rank, target modules, alpha; QLoRA if you want to train on a consumer GPU; full fine-tune only if you have the budget.
Training loop: HuggingFace Transformers + PEFT + TRL as the canonical stack; Axolotl or llama-factory as convenience wrappers; Unsloth if you want 2-4× more speed on consumer GPUs.
Hyperparameter sweep: W&B Sweeps, Optuna, Ray Tune.
Checkpointing and resumability: save every N steps, resume from failure.
Promotion: the adapter is promoted to the registry after passing the next stage (Eval).

The three modes of Tune

The detail of the table we saw in the landscape:

Supervised fine-tuning (SFT) with LoRA/QLoRA. You collect (prompt, ideal-response) pairs and apply SFT with cross-entropy loss. The simplest option. The rule of thumb: 300-3,000 well-curated examples are usually more useful than 50,000 noisy ones.

DPO (Direct Preference Optimization) and RLAIF. Instead of “ideal-response”, you collect (prompt, good_response, bad_response) triples and train the model to prefer the good one. More stable than classic RLHF, same objective. It is what most teams use when they go beyond SFT.

Agent training (RFT / Reinforcement Fine-Tuning, pure RLHF). For cases where the model needs to learn multistep trajectories: when to choose tool A over B, when to ask for confirmation, how to decompose a large task. Far more expensive and complex. OpenAI’s work with RFT set the pattern in 2024-2025; in 2026 it is coming out of the experimental phase.

RAG as an alternative to Tune: although conceptually it is another stage (it lives in Data + Deploy), functionally it competes with fine-tuning for many cases. The 2026 verdict: hybrid is the default (60% of deployments), fine-tune for behaviour plus RAG for volatile knowledge.

Tools

Aspect	Tools 2026
Base framework	HuggingFace Transformers, PEFT, TRL
Convenience wrappers	Axolotl, llama-factory
Consumer speed	Unsloth (2-4× speedup on RTX GPUs)
Distributed training	DeepSpeed, FSDP, NeMo Framework
Experiment tracking	MLflow, W&B, ClearML
Adapter registry	Private HuggingFace Hub, MLflow registry
Hyperparameter	W&B Sweeps, Optuna, Ray Tune

Traps

Catastrophic forgetting: very aggressive SFT destroys the model’s general capabilities. Keep a small % of the original dataset or use regularisation.
Overfitting to the golden dataset: the model learns to memorise the eval set. Keep a holdout test set that nobody on the team looks at until the final release.
Train/serve skew: prompts in training with a different format from production. The same chat template in both.
LoRA rank too high: it looks like it improves metrics but it inflates the adapter with no real benefit. Start with r=8 or r=16; raise it only if there is evidence.

Stage 3 — Eval: validating before promoting

Operational sub-tasks

Covered in depth in Evals: the layer after tracing. A structured summary for the pipeline:

Golden dataset curation: 100-500 examples as a minimum, actively maintained with incident cases.
Evaluators: heuristic (regex, length), semantic (embeddings), LLM-as-judge (G-Eval), human (golden labels).
Running in CI: block the merge if critical metrics drop by more than X%.
Running on the platform: over sampled production traffic, persist results, detect long-term regression.
Judge calibration: 85-90% agreement with humans before accepting the judge as productive.
Eval gates: explicit thresholds per metric (faithfulness > 0.85, relevancy > 0.80, and so on).

Tools

CI gates: DeepEval (Apache 2.0, pytest-style), Promptfoo (MIT, CLI), Ragas (RAG-specific), Inspect AI (safety/capability).
Platform: Langfuse (MIT, complete suite), LangSmith (LangChain), Phoenix (ELv2, OTel), Braintrust.
Judges: GPT-4 (expensive but the reference), Claude 3.5 Sonnet, Prometheus (OSS, 0.897 correlation), JudgeLM.

Traps

Aged golden dataset: if it is not updated, it stops reflecting production.
Contaminated judge: the judge knows the dataset (it appeared in its training).
Insufficient sample size: fewer than 50 examples makes differences look like noise.
Runaway costs: G-Eval with GPT-4 over many cases costs thousands of USD a month.
Forgetting the segment: an average of 0.85 can hide a 0.55 in German.

Stage 4 — Deploy: putting the model into production

Operational sub-tasks

Covered in depth in vLLM on Kubernetes and LLM operators on K8s. A summary for the pipeline:

Runtime selection: vLLM (default), SGLang (agents with high prefix caching), TensorRT-LLM (pure latency), llama.cpp (edge).
Operator selection: vLLM Production Stack, KServe, OME (LMSYS), NVIDIA Dynamo, llm-d (CNCF).
Serving configuration: --tensor-parallel-size, --kv-cache-dtype=fp8, --enable-prefix-caching, --enable-chunked-prefill, --gpu-memory-utilization=0.92.
Routing between models: LiteLLM as the multi-provider abstraction.
Release strategy: canary (1% → 10% → 100%), blue-green (all or nothing with fast rollback), shadow (eval in parallel without affecting users).
Autoscaling with LLM metrics: KEDA + Prometheus over vllm:num_requests_waiting or equivalent.
Gateway / Inference Extension: Gateway API Inference Extension once it is GA.

Dominant tools

Serving engines: vLLM, SGLang, TensorRT-LLM, llama.cpp, MLX.
Operators: OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, KServe.
Routing: LiteLLM (100+ providers), OpenRouter (managed), LangChain Router.
GPU primitives: NVIDIA GPU Operator, LeaderWorkerSet (LWS) for multi-pod tensor parallel, KEDA for autoscaling.

Traps

Naïve rolling update that cuts sessions: maxUnavailable: 0, maxSurge: 1 and terminationGracePeriodSeconds: 120+.
A short readiness probe that kills pods while they load: startupProbe with failureThreshold: 60.
HPA on CPU% with no LLM metrics: vLLM batches internally, one replica serves dozens. KEDA on queue depth.
Unquantised KV cache: --kv-cache-dtype=fp8 is almost always worth it.
Tensor parallel on GPUs without NVLink: all-reduce saturates PCIe and throughput collapses.

Stage 5 — Observe: seeing what happens in production

Operational sub-tasks

This is the stage we have covered most deeply in previous series: the whole eBPF series (4 posts) and the whole post-tracing series (4 posts) deal with sub-tasks of Observe. A structured summary:

Tracing: OpenLLMetry/Traceloop, Langfuse, Phoenix, LangSmith. Spans with OTel GenAI semantic conventions (gen_ai.*, mcp.*).
Metrics: Prometheus + Grafana. TTFT, TPOT, throughput, queue depth, KV cache usage, cost per tool.
Active guardrails (not just eval): NeMo Guardrails with 5 types of rails, multimodal Llama Guard 4, Llama Prompt Guard 2 (86M/22M), LLM Guard.
eBPF observability (zero-instrumentation): Hubble (network), Tetragon (process/syscall), AgentSight (LLM agent with SSL uprobes + stdiocap MCP).
eBPF in the local engine (inference): ProfInfer-style with uprobes in llama.cpp / vLLM / libcudart.
Drift detection: Evidently AI, NannyML, WhyLabs. KS, PSI, MMD over embeddings.
MCP observability: OpenTelemetry GenAI MCP semantic conventions, trace propagation via params._meta, a centralised MCP Gateway.

The four mandatory metrics

Out of everything covered, the four any minimal dashboard must have:

TTFT p50/p95 (time to first token) — what the user perceives.
TPOT p50/p95 (time per output token) — streaming speed.
Throughput (aggregate tokens per second) — capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator.

To these you add, per domain:

For RAG: faithfulness rolling mean, retrieval hit rate.
For agents: tool call accuracy, multi-step task completion.
For multi-tenant: cost per tenant, p95 latency per tenant.

Traps

Cardinality in Prometheus: metrics carrying every K8s label blow up.
Tracing with no sampling: storage grows out of control.
Guardrails permanently in monitoring mode: they never reach enforce.
Drift with no alerts: you spot drift on the dashboard once a month; meanwhile the problem has been running for weeks.
OTel with no propagation: MCP, Tetragon and AgentSight spans left disconnected.

Stage 6 — Retrain: closing the loop

Operational sub-tasks

This is the stage most neglected in GenAI projects. Closing the loop turns LLMOps into a living practice; not closing it leaves it as a project that ages.

Explicit feedback: thumbs up/down in the UI, annotations by power users, forms for “what went wrong”.
Implicit feedback: anomalous latency, abandonment rate, user retries, aborted sessions.
Incident triaging: classify incidents by root cause (model issue, retrieval issue, prompt issue, infra issue).
Dataset enrichment: add to the golden dataset the cases where the system failed, with the correct answer labelled by a human.
Retrain cadence: quarterly by default, incident-driven when a problematic pattern goes over a threshold.
Promotion: the new model/adapter goes through the Tune → Eval → Deploy stages, with eval gates that compare against the model in production.

The two cadences

Scheduled retrain (quarterly or half-yearly): an established process. It lets you plan capacity, budget and risk. The default.

Incident-driven retrain: when a serious incident (detected drift, a failing segment, a prompt injection attack) crosses a threshold, a mini-cycle fires. More expensive but necessary for critical cases.

Dominant tools

Annotation and feedback collection: Langfuse (built-in UI), Argilla (OSS), Label Studio.
Dataset enrichment: pipelines in Airflow or Argo Workflows.
Triaging: Langfuse dashboards with filters on traces with a low eval.
Promoting candidates: MLflow model registry stages.

Traps

An open loop: production does not inform the dataset; the model never improves.
Human feedback gets lost: thumbs down with no structured capture channel.
Undefined cadence: “we retrain when it is needed” → nobody ever retrains.
No holdout test set: the golden dataset is enriched with the same cases used to evaluate; eval measures memorisation.
Promotion with no gates: the new model enters production without passing the checks the previous models passed.

The complete cycle: how the stages fit together

Now that we have seen each stage separately, the key insight is how they hook into each other. Five emergent properties of the cycle:

1. Data is the raw material of every stage. Tune reads from the golden dataset. Eval reads from the eval dataset. Deploy reads from the RAG (vector store). Observe produces new data. Retrain creates new datasets. The Kafka log is the gospel of the whole system (post 2 in the series).

2. Eval is the bidirectional gatekeeper. Before Deploy: it blocks the release if the model regresses. After Observe: it feeds Retrain by identifying the worst-scoring cases. The quality of the eval determines the quality of the whole cycle.

3. Observe feeds Retrain and Eval simultaneously. Traces produce metrics for Observe; problematic traces are annotated and go to the dataset; the new cases enrich the eval golden set. Observe is the operational source of truth.

4. The cross-cutting components (the grey band on the map) are not a stage, they are an infrastructure. OpenTelemetry, prompt versioning, MCP gateway, model gateway, schema registry. Badly configured, every stage suffers separately. Well configured, the stages integrate without friction.

5. The cycle is not strictly sequential, it is concurrent. At any given moment the system has: requests being served (Deploy + Observe), a new version in training (Tune), continuous eval in CI (Eval), data arriving from CDC (Data), incident analysis (Retrain). Every stage is alive at once.

Cross-stage traps: things that break the whole system

There are errors that do not belong to a stage but to the interfaces between stages. The most common ones:

Train/serve skew

The exact format of the prompt in training is different from the one in production. The result: the model trained to answer <|im_start|>user\n...\n<|im_end|> receives User: ...\nAssistant: in production and performs worse. Solution: extract the chat template into a shared library used by the Tune pipeline and the Deploy one.

An eval that does not reflect production

Your golden dataset is careful questions; production is real questions with typos, mixed languages, and so on. Eval passes at 95%, production performs at 70%. Solution: continuously enrich the golden set with real samples.

Drift with no response pipeline

You spot drift on the Observe dashboard; nobody has a defined workflow for what to do. Solution: every drift alert must have a clear runbook: investigate, classify, act (retrain, adjust the prompt, widen the retrieval).

Schema break cascade

You change the schema at the OLTP source; Debezium reflects it; the Flink job breaks; the embedded topic stops updating; the vector store ages; the RAG answers over old data. Three stages affected by one change in Data. Solution: mandatory backward-compatible schema evolution, contracts between producers and consumers.

No observability of the pipeline itself

The LLMOps pipeline is a complex system. If it has no observability of its own (how long training takes, how many jobs fail, how many re-embeddings run), debugging failures is an exercise in spelunking. Solution: OTel over the pipeline itself, not just over the LLM calls.

Invisible vendor lock-in

Pipelines written against LangChain, prompts stuck in LangSmith, embeddings in Pinecone, the model in OpenAI. Migrating is a months-long project. Solution: LiteLLM and OpenLLMetry abstractions, vendor-neutral from the start.

What comes in the following posts

Post 4 — PostgreSQL + Qdrant in the ingestion stage — the first post that applies the “you are here” pattern over the Data stage. Synchronisation patterns (outbox + CDC), microservices architecture, deployment manifests.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / runtime alignment, continuous fine-tuning in depth, edge LLMs.
In any later post in this or future series, the “you are here” mini-map will tell you which stage of the cycle the topic fits into. If you read a post about quantization, you will know you are in Deploy. If you read one about evaluator ensembles, you will know you are in Eval. If you read one about RAG over Iceberg, you will know you are in Data.
If you want to see the whole pipeline in action following a single real request, the synthesis post Anatomy of an LLM request in production does exactly that: it rewinds a request back to the data that trained it 90 days earlier and follows it forward to the feedback that will reappear in the next Retrain cycle, crossing the six stages and the two cross-cutting components in one coherent story.
If what interests you is comparing how each stage is built in open source against the hyperscalers, the post The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure makes the vertical cut: for every stage, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, gap identification and the multi-tenant chatbot ported to an AWS stack as a concrete example.
If you want the OSS toolbox piece by piece, the post The OSS catalogue for LLMOps in six stages: entry by entry does the zoom in: ~150 words of description per core tool (what it does, how it differs from its alternatives, licence and governance, typical gotcha), a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.
If what you are missing is the common vocabulary that runs through the six stages, the post Ontologies and knowledge graphs in LLMOps walks the six stages from the perspective of shared formal nomenclature: how TBox + ABox + SHACL + SKOS change the operation of Data, Train/Adapt, Eval, Deploy, Observe and Govern, the GraphRAG 2026 landscape (Microsoft GraphRAG v2, LightRAG, HippoRAG 2, KAG/OpenSPG), vertical ontologies actually deployed (FIBO, SNOMED CT, schema.org, ENS, EU AI Act) and an open source stack with its licensing caveats (Neo4j Community GPLv3, KuzuDB upstream archived in October 2025).

References

Foundations:

Per stage (entries from the blog series):

Data: RAG over Kafka — technical architecture and the dedicated post on data versioning with DVC and lakeFS — the four artefacts to version separately (training, RAG corpus, golden eval, enriched retrain), schema contracts, end-to-end lineage from dataset to trace, and why a golden set without a strict holdout measures memorisation.
Tune: partly covered in Landscape 2026; a deeper treatment in post 4 if continuous fine-tuning is chosen.
Eval: Evals: the layer after tracing.
Deploy: vLLM on Kubernetes and LLM operators on K8s.
Observe: the whole eBPF series and the whole post-tracing series.
Retrain: the dedicated post on how to close the loop — feedback capture (explicit + implicit), triage by root cause, dataset enrichment with human annotation (Argilla / Label Studio), scheduled vs incident-driven cadences, governed promotion with eval gates.

Cross-cutting components:

Prompt versioning: the dedicated post with Langfuse and MLflow Prompts — the three-primitive pattern (immutable version, mutable label, cache), eval gates on promotion, and per-request traceability.
MCP: deep MCP observability.
Drift detection: eBPF + drift detection.
Local inference: PagedAttention deep dive, KV cache.

Frameworks and tools referenced:

MLflow, W&B, Kubeflow, ZenML, BentoML, Metaflow.
HuggingFace PEFT, TRL, Axolotl, Unsloth.
DVC + lakeFS.
Langfuse, Evidently AI, Phoenix.
vLLM Production Stack, KServe, OME, NVIDIA Dynamo, llm-d.

Architecture on lo0 — Tech Blog

One big replica or many small ones: the decision that defines your platform

TL;DR

The analogy

The two architectures in vLLM

Architecture A: TP=4, one replica

Architecture B: TP=2 × 2 replicas

Why TP=4 has higher individual latency than TP=2

The crossover point: when TP=2×2 beats TP=4×1

The three implications nobody mentions

1. Fault tolerance

2. Autoscaling granularity

3. Quality degradation under load

Measuring the crossover point with OTel

The decision by workload profile

Kubernetes configuration for both architectures

See also

References

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

TL;DR

You are here: the seven phases and their dependencies

The analogy: the expedition to an eight-thousand-metre summit

F0 — Hardware in the room: base camp

F1 — Bare metal: the operational base camp

F2 — Kubernetes cluster operational

F3 — GitOps and infrastructure observability

F4 — Identity, certificates, secrets, policies

F5 — GPU platform with LLM-aware observability

F6 — LLM stack in production

The maths that matter: relative weight of effort per phase

Final diagram: the complete deployment schedule

Typical planning mistakes

Applied to typical on-premise hardware: 4×H100 SXM

What we have not covered (upcoming posts)

See also

References

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

TL;DR

You are here: the five levels at a glance

The analogy: from the street stall to the starred restaurant

Level 0 — Chaotic: the server with Linux and nothing else

Level 1 — Repeatable: an operational Kubernetes cluster

Level 2 — Defined: the cluster rebuilds from git

Level 3 — Managed: identities, certificates, secrets and policies

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

The maths that matter: what skipping a level costs

Final diagram: the complete ladder with pieces

Typical design decisions that break progress

Applied to typical on-premise hardware: a 4×H100 SXM cluster

What we have not covered (upcoming posts)

See also

References

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

TL;DR

You are here: the seven layers seen from above

The analogy: the office building with shared services

The hook: the log that lied for six days

Layer 1 — Gateway: the client SDK must not couple to the engine

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

Layer 4 — Vector store + relational data + storage: what actually holds the state

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

Layer 6 — GitOps control plane: the only legitimate authority

Layer 7 — Dependency tracking: the layer the incident exposed

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

Final diagram: the complete connected stack

Typical design decisions that break the stack

Applied to typical on-premise hardware: a 4×H100 SXM cluster

What we have not covered (upcoming posts)

References

Relevant official documentation

The six-stage LLMOps pipeline: global architecture and a deep dive into every component

TL;DR

The global architecture: the master map

Stage 1 — Data: ingestion, transport, versioning, indexing

Operational sub-tasks

Dominant tools

Design decisions

Traps