Synthesis on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

Fri, 22 May 2026 16:00:00 +0200

TL;DR

Across several series the blog has laid out the pieces that hold up an LLM system in production: the Data stage (dataset versioning, ingestion and vector stores, RAG over Kafka), the Tune stage (continuous fine-tuning), the Eval stage (evals as the layer after tracing, guardrails and safety), the Deploy stage (KV cache, PagedAttention, disaggregated serving, multi-tenant GPU cluster, vLLM on Kubernetes, LLM operators on K8s), the Observe stage (tracing with AgentSight, MCP observability, eBPF + drift), the Retrain stage (closing the feedback → dataset → adapter loop), and the cross-cutting components (prompt versioning and data versioning). What is missing is joining them up: seeing a single request cross every piece in order, in one coherent story. That is what this post does. We take a specific request from a multi-tenant support chatbot, rewind it back to the data that trained the adapter serving it today, follow it forward through serving, watch it reach the feedback store when the user marks a thumbs-down, and leave it as the seed of the next quarterly retrain cycle. The route works as a mental map and as an integrator’s guide: the system does not hold up if a single one of the seven pieces (six stages + two cross-cutting) is broken or absent. The practical lesson of the tour is not a new one. It is that everything is connected, that local measurements lie when taken in isolation, and that the real cost of not operating one stage well is paid by another stage further down the line.

You are here: every stage at once

Unlike previous posts, where the mini-map marked a single active box, this one covers the whole pipeline. It is the only post on the blog that activates the six stages and the two cross-cutting components simultaneously, because we follow a real request that crosses all of them.

The analogy: forensic analysis of a request

When an air accident happens, the forensic analysis does not limit itself to the last seconds of the flight. The investigation team rewinds to the maintenance of the previous six months, the manufacturer’s protocols, the pilot’s record, the weather briefing, the controller’s decisions, the incident history on the same model. The conclusion is rarely “the wing broke”; it is “the wing broke because an inspection protocol written a certain way did not detect microcracks that the 2014 calculation model did not consider critical and that were critical beyond a certain fatigue cycle”.

When an LLM request in production fails or succeeds, there is also a long causal chain behind it. The answer the user sees is the last frame; what determined it starts months earlier and branches across six operational stages. If you only look at the last frame, you attribute the result to the model. If you look at the whole chain, you see that the model is one of twelve factors and rarely the most important.

This post does that forensic analysis, but the other way round: instead of starting from a failure and rewinding, we start from a specific request that works and break down what had to happen for it to work, and what will happen to it afterwards. It is a guided tour, not an incident diagnosis. But the mental discipline is the same: no stage is autonomous, and understanding the system means understanding the bridges between stages, not just the boxes.

The scenario: a multi-tenant support chatbot for regulated customers

For the tour we use a concrete realistic scenario, representative enough that the observations carry over to most serious deployments in May 2026. It is an LLM customer support SaaS product, sold to several corporate customers (multi-tenant) in regulated sectors (banking, insurance, health). The product:

Accepts natural-language questions through a chat embedded in the customer’s website.
Retrieves relevant fragments from the customer’s internal knowledge base (product documents, contractual terms, FAQs).
Answers citing the fragment used, in the user’s language, with a tone adapted to the brand.
If the question requires an action (cancel, change plan, open a case), it invokes a tool from the catalogue exposed via MCP, with user confirmation.
It is contractually obliged not to invent customer data, to respect the tenant’s style guide, and to meet ENS / NIS2 compliance on data in the EU.

The infrastructure is on-premise over Kubernetes (RKE2 with the Cilium CNI, in the pattern covered in Cilium BGP and CNI networking). The GPU plane is a generic 4×H100 SXM cluster (80 GB each, NVLink) for serving and production training, plus an RTX 4090 (24 GB) for development, offline evals and ad-hoc re-embeddings.

The request we are going to follow:

User (end customer, Spanish, premium plan with an insurer): “How do I cancel my premium subscription? I have been paying for three months and I have not used half of the cover.”

At first sight, a simple question. Let us see what is behind it.

Step 0 — The frame: the answer the user sees

At 11:47:32 on a Tuesday, the user sends the message. 4.2 seconds later, this appears in the UI:

To cancel your premium subscription, follow these steps: (1) Sign in to the customer portal with your ID number and password. (2) Go to My policies → Premium Life 2024. (3) Press Request cancellation. (4) Select the reason (in your case, “not using the product”) and confirm. You will receive a confirmation email within 24-48 h.

Before you cancel, let me remind you that your plan includes a free annual medical check-up and 24/7 travel assistance, cover that is lost on cancellation. If you like, I can help you review your use of the cover over the past three months to see whether switching to a more tailored plan suits you better than cancelling.

Source: [Cancellation policy, section 4.2 of the 2024 General Terms]

After reading it, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”.

That sequence, question, answer, thumbs-down with structured feedback, is the last visible frame. Let us go backwards to understand what had to happen for the answer to come out that way.

Rewinding backwards: what was already in place before the request

Before the user typed, the system already had a model loaded in serving, an active prompt labelled production, an up-to-date vector index, a versioned dataset from the last fine-tuning, and a golden eval set that validated the promotion. Each of those artefacts got there through a process. We go through four jumps backwards.

t = −90 days — The previous Retrain stage closes the earlier cycle

Three months ago, during a quarterly Retrain cycle, two things happened. The first: the support team reviewed the feedback accumulated over the previous six months and saw a pattern, the model answering premium users in an excessively formal tone, with users reporting that it “feels robotic”. The second: a one-off incident (a customer cancelling over an answer perceived as curt) triggered an incident-driven mini-cycle.

The process, covered in detail in the Retrain post, followed five sub-processes:

Feedback capture — explicit thumbs-down plus implicit feedback (abandonments, retries) accumulated in a feedback_signals table in Postgres, all of them with a trace_id that allows rewinding to the exact context.
Triage by root cause — the “curt tone” incident cluster was categorised as a prompt issue (it was not the model answering badly, it was the system prompt asking for an overly formal register). One sub-cluster was a model issue (in some cases the model dug its heels in even with a warmer prompt).
Dataset enrichment — the team manually annotated 280 cases where the model was too curt, labelled with the reference answer (“how it should have answered”). Double annotation on the critical 20%; cases with a quality score below 4 were left out.
Cadence decision — the incident was treated as incident-driven; the rest of the quarterly Retrain followed the calendar.
Promotion — the new customer_support_v7 adapter went through eval gates against customer_support_v6, a 5% canary for a week, and was promoted when the golden set metrics showed a stable improvement in the “tone / clarity” segment with no regressions elsewhere.

The result: the adapter active in production when the user sent the Step 0 request is customer_support_v7, trained on the enriched dataset enriched_retrain_2026_q1 version 3, with double lineage back to the original incident.

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

Immediately after Retrain, the Data stage of the LLMOps pipeline does its work. Three critical operations, covered in detail in the data versioning post:

Immutable versioning of the enriched dataset with DVC, sha256 hash propagated to the registry. The identifier (enriched_retrain_2026_q1, v3, sha256:9af...) becomes the luggage tag that will travel through the coming stages.
Schema contract validated by CI: every row meets the JSON Schema of the entry the trainer expects (example_id, input.user_query, input.retrieved_context, expected_output, rubric, segment, difficulty). A validation fails in CI if any row breaks the contract.
Holdout segregation check: the normalised sha256 hash of each input is compared against every hash in the active golden eval set (customer_support_golden_v12). Zero overlaps = the dataset does not contaminate the eval. Had there been a single one, CI would have blocked the merge.

In parallel, the RAG corpus (product manuals, FAQs, general terms of the insurer tenant) is kept alive. The ingestion pipeline keeps capturing changes from the customer’s CMS: a new section of the cancellation policy was modified in February and reindexed in Qdrant. As the post about RAG over Kafka explains, the corpus is not retrained with every change: only the delta is re-embedded, and lakeFS keeps a branch of the embeddings bucket with the new version. The branch is merged into main when recall@10 over a representative set of queries stays above the threshold (0.78 in this system).

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

Three weeks after closing the dataset, training of the new LoRA adapter starts. As the continuous fine-tuning post details, the production pattern in 2026 avoids retraining the base model, which is expensive, slow and irreversible, and favours a LoRA adapter over a stable base model (in this system, Llama 3 70B-instruct quantised to INT8 for serving). The training:

Runs over 4 of the H100s (NVLink, tensor parallel) for ~6 hours.
Uses transformers + PEFT + bitsandbytes, monitored by MLflow.
Every step records the dataset_id, dataset_version and dataset_hash as an input artifact in MLflow.
The output, a customer_support_v7.safetensors file of ~280 MB with the LoRA weights, is uploaded to MinIO with its own hash, and MLflow records model_id, model_version, parent_dataset.

At this point the lineage chain is closed for this leg:

enriched_retrain_2026_q1, v3, sha256:9af...
│
▼
mlflow run train, run_id: 0xa721...
│
▼
customer_support_v7, sha256:5c1...

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

The freshly trained adapter is not promoted. It goes through an eval suite covered in detail in the post about evals. The golden eval set, customer_support_golden_v12, 850 human-curated examples with an inter-annotator kappa of 0.81, is run against two models: the candidate v7 adapter and the v6 currently in production. The metrics:

Metric	v6 (prod)	v7 (cand.)	Threshold
Faithfulness to the RAG fragment	0.87	0.89	≥ 0.82
Toxicity (low is good)	0.012	0.011	≤ 0.02
“Warm but professional” tone (judge LLM)	0.71	0.84	≥ 0.78
Format compliance (structured markdown)	0.94	0.93	≥ 0.90
Helpful-but-not-pushy (judge LLM)	0.66	0.79	≥ 0.75
Latency p95 (ms)	2,840	2,910	≤ 3,500

To this is added the guardrails and safety suite covered in the guardrails post: jailbreak resistance, PII leakage detection, prompt injection over MCP tools. The v7 improves on safety in two metrics and ties on the rest.

The v7 enters a 5% traffic canary for 7 days, under close monitoring. At the end of the canary, the online metrics confirm what the offline ones anticipated: an improvement in tone and helpfulness, equivalent latency, no new failure modes. Promotion approved. The v7 moves to the production label.

t = −31 days — The Deploy stage: the v7 adapter enters serving

The customer_support_v7 adapter is promoted to the serving cluster. Three pieces covered in separate posts come into play.

vLLM as the inference engine. The engine lives on Kubernetes, deployed via a dedicated Operator, as the post about LLM operators and the post about vLLM on K8s explain. The operator is responsible for detecting the new adapter in the registry, hot-loading it without restarting the engine (a native vLLM capability with --enable-lora), and directing traffic based on the label.

Disaggregated serving. As the post about disaggregated serving details, the system separates prefill (compute-intensive, throughput-bound) and decode (memory-intensive, latency-bound) into different GPU pools. The user’s request, when it arrives, prefills in a specialised pod and decodes in another, communicating over NVLink plus a shared KV cache fabric.

Multi-tenant GPU cluster. The H100 cluster serves several tenants, not just the insurer from Step 0. As the post about the multi-tenant cluster explains, isolation is realised on four planes: the Kubernetes namespace, ACLs over adapters (only the tenant’s namespace loads its LoRAs), partitioning of the KV cache per tenant (one tenant cannot read another’s cached prefixes), and a tokens-per-minute quota enforced at the gateway.

Synchronised prompt registry. The product’s system_prompt lives in Langfuse with the production label. The active version is customer_support_system_prompt, version 12. The gateway reads the prompt from Langfuse on the request path (with a few seconds of cache so it does not hammer the registry). Detailed in the prompt versioning post.

The result at t = −31 days: the combination (adapter v7, prompt v12, golden v12) is active and served. The system is ready for the request that will arrive 31 days later.

Moving forward: the user’s request crosses the system

Back to Step 0: 11:47:32 on a Tuesday. The user presses Enter. We go in real time, in milliseconds.

t = 0 ms — Entry through the gateway

The user’s browser POSTs to chat.aseguradora-ejemplo.com/api/chat. The traffic crosses the edge load balancer and enters the SaaS product’s API gateway. The gateway:

Authenticates the user’s JWT (an end customer of the insurer tenant).
Extracts the tenant_id and checks that its tokens-per-minute quota is not exhausted.
Resolves which model_id, adapter_id and prompt_id correspond to this tenant and product. In this case: llama-3-70b-int8 + customer_support_v7 + prompt label production.
Builds a unique trace_id (W3C TraceContext, propagable to OTel) and starts a root span.

At 8 ms, the gateway passes the request to the prefill pool.

t = 8 ms — Pulling the versioned prompt

Before serving, the OpenAI-compatible client the engine uses internally pulls the active system prompt. As the post about prompt versioning details, the pattern is:

prompt = prompt_registry.pull(
 name="customer_support_system_prompt",
 label="production", # now pointing at v12
)
# A local 30 s cache reduces the round-trip to 0.1 % of requests

The OTel span of the prompt pull carries the attributes gen_ai.prompt.id = customer_support_system_prompt, gen_ai.prompt.version = 12, gen_ai.prompt.label = production. They are propagated to all children.

t = 12 ms — RAG retrieval

The system needs context from the tenant’s knowledge base. It runs:

query_embedding = encoder.encode(user_query)
chunks = qdrant.search(
 collection=f"tenant_{tenant_id}_kb_v3",
 vector=query_embedding,
 limit=4,
 score_threshold=0.72,
)
reranked = reranker.rerank(user_query, chunks, top_k=2)

At 38 ms, the reranker returns two fragments: one from the Cancellation policy, section 4.2 and another from Premium plan benefits, section 2.1. As the post about PostgreSQL + Qdrant details, the tenant’s corpus is kept isolated by collection and ACL: no tenant can read another’s chunks.

t = 40 ms — Building the final payload

The engine composes:

[system_prompt v12]
+ [retrieved context: 2 chunks]
+ [short session history: 1 previous turn]
+ [user query]

Total: ~1,850 tokens of context. The OTel span records gen_ai.request.input_tokens = 1850, gen_ai.request.model = llama-3-70b-int8, gen_ai.request.adapter = customer_support_v7.

t = 45 ms — Prefill

The payload enters the prefill pool. The GPU processes the 1,850 tokens in a single parallel pass, computing for each token its K and V vectors (attention key and value). Those vectors are materialised as the KV cache, covered in detail in the KV cache fundamentals post. The resulting cache takes up ~120 MB of VRAM in INT8.

Here a key optimisation appears: the v12 system prompt is cached in the prefill pool (prefix caching, covered in the post about PagedAttention). Since the system prompt is the same for this tenant, the first ~500 tokens of the context are not recomputed: they are read from the prefix cache. That reduces the effective prefill from 1,850 tokens to ~1,350 tokens, saving ~270 ms of compute.

At 580 ms (effective prefill), the TTFT (time to first token) is ready. The first token leaves for the decode pool.

t = 580 ms — Decode (streaming)

The decode pool receives the prefilled KV cache and starts generation token by token. As the post about disaggregated serving details, the prefill/decode separation is what lets a multi-tenant system keep TPS stable: the decode pool is sized to sustain thousands of sessions decoding in parallel at a low cost per token, while the prefill pool is sized for bursts of short TTFT.

Generation at ~80 tokens per second. The answer will be ~290 tokens. Total decode time: ~3.6 s. Streaming: the user starts seeing words from t = 580 ms.

While decode advances, the engine emits child spans on every iteration with gen_ai.response.tokens_generated, gen_ai.response.cache_hit_ratio, gen_ai.response.cumulative_latency. The post about AgentSight and the post about MCP observability with OTel cover the detailed instrumentation of this layer.

t = 4,200 ms — Complete answer, root span closed

Generation finishes. The engine closes the root span with gen_ai.response.completion_tokens = 290, gen_ai.response.finish_reason = stop, gen_ai.response.total_latency_ms = 4200. The user sees the final answer. The session is ready for a next turn or for the user to click thumbs-up/thumbs-down.

By this point, every active stage has taken part:

Data (pre-existing): the indexed RAG corpus, the dataset that trained the adapter, the golden set that validated it.
Tune (pre-existing): the v7 adapter trained 45 days ago.
Eval (pre-existing): the gates that approved the promotion.
Deploy (at this very moment): vLLM + disaggregated + KV cache + multi-tenant.
Observe (at this very moment): the OTel spans emitted to Langfuse + Tempo, the metrics to Prometheus.
Retrain (about to activate): the feedback the user will leave in 15 seconds.

In parallel: Observe is watching

While the request happens, several pieces of Observe run in parallel and leave a structured trail.

OTel tracing. Every span (gateway, prompt pull, retrieval, prefill, decode) travels to Langfuse and to an OTel collector that forwards them to a backend (Tempo / Jaeger). The unique trace_id links all the spans. As the post about tracing with AgentSight details, end-to-end propagation is the main enabler of post-incident debugging: without it, you cannot reconstruct what happened three weeks later.

Runtime metrics. The engine emits Prometheus metrics per interval: gpu_utilization, kv_cache_usage, tokens_per_second, queue_depth, prefill_latency_p95, decode_latency_p95. The metrics are not tied to a trace; they are aggregated by tenant and service.

Online LLM-as-judge. A configurable percentage of answers (2% in this system) is also run through a judge LLM in the background, which scores the answer against a simple rubric (correct / partial / incorrect + a tone score). The judge does not block the answer to the user; it feeds the dashboard.

Statistical drift. In parallel, a slower pipeline computes drift over the distribution of inputs and outputs. As the post about eBPF + drift explains, low-level monitoring (latency, error rate per endpoint) is complemented with statistical drift detection (KS test, embedding distance) that spots when “something is wrong” before a thumbs-down confirms it.

Safety and guardrails monitor. The guardrails post describes the layer that watches for jailbreak attempts, PII leakage and prompt injection via MCP tools. In this case, none of them fires.

All these pieces operate continuously, not per request. But this particular request left its trail in every one of them.

The feedback: the loop closes

Fifteen seconds after reading the answer, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”. That apparently trivial gesture triggers an important sequence.

Insertion into feedback_signals

As the post about Retrain details, the thumbs-down is persisted as a structured row in a Postgres table:

INSERT INTO feedback_signals (
 signal_id, trace_id, request_id, signal_type, signal_value,
 prompt_id, prompt_version, model, user_segment, occurred_at
) VALUES (
 gen_random_uuid(),
 '4f5...', -- the trace_id from Step 0
 'r-22a...', -- request_id
 'thumbs',
 '{"vote":"down","reason":"too pushy","text":"I only wanted to know how to cancel"}',
 'customer_support_system_prompt',
 12,
 'llama-3-70b-int8+customer_support_v7',
 'premium-es',
 '2026-05-19T11:47:51+02:00'
);

With this, the row is linked by trace_id to everything that happened: prompt v12, retrieved context, complete output, latency metrics, judge score (0.82 in this case, considered good by the judge, though the human disagrees).

Triage by root cause

The MLE team runs triage the next morning. Combining heuristic rules, LLM-as-classifier and human review:

The signal is not a model issue: the model answered correctly to the prompt it received.
It is not a retrieval issue: the retrieved chunks were the right ones.
It is not an infra issue: latency was normal.
It is a prompt issue: the v12 system prompt instructs the model to “offer alternatives before processing destructive actions”. That instruction generates the “salesy tone” in some contexts.

The incident accumulates with others from the month in the “salesy tone” cluster. When the cluster crosses a threshold (typically 30-50 incidents of the same type or a percentage of the total), it will enter an incident-driven mini-cycle or wait for the quarterly Retrain, depending on the size.

The next cycle picks it up

Three months later, in the next quarterly Retrain, this feedback is one of many that will motivate two changes:

A new prompt version v13 with an adjusted instruction: “offer alternatives only if the user does not express a clear intention to cancel”.
A possible reinforcement of the adapter with cases of a more direct tone for premium-es. If the cluster justifies it.

The v13 will go into its own eval gate. The golden set will grow with cases where the correct tone is “direct, not salesy”. The v8 of the adapter (if it arrives) will retrain over the enriched dataset enriched_retrain_2026_q2 that already contains this annotated case.

The loop closes. The Step 0 request has contributed to the version of the system that will serve another user three months later.

What goes in each trace: identity and traceability

If the reader looks at the seven identifiers that are everywhere in this route, they see the network of identities that makes all of the above possible. It is the identity infrastructure of an LLM system in production:

trace_id 4f5... (unique per request)
request_id r-22a... (idem)
prompt_id customer_support_system_prompt
prompt_version 12
prompt_label production
dataset_id enriched_retrain_2026_q1
dataset_version v3 (sha256:9af...)
model_id llama-3-70b-int8
adapter_id customer_support_v7 (sha256:5c1...)
deployment_id d-prod-7b
schema_version 3.2
tenant_id aseguradora-ejemplo
user_segment premium-es
golden_set_id customer_support_golden_v12

If a single piece of that set is missing or does not propagate, the chain breaks. The next incident investigated will land in “we cannot rewind to the origin because the system did not record it”. That is why the cross-cutting components, prompt versioning and data versioning, are not luxuries: they are the connection without which the other six stages operate blind.

Synthesis diagram: how the pieces fit

 ┌─────────────────────────────────────────┐
│ User (end customer, B2C) │
└─────────────────┬───────────────────────┘
│ chat msg + JWT
▼
┌─────────────────────────────────────────┐
│ Edge LB + WAF + Cilium CNI │
└─────────────────┬───────────────────────┘
│ HTTPS, internal mTLS
▼
┌─────────────────────────────────────────────────┐
│ API Gateway (auth, quota, model routing) │
│ - Resolves tenant → model + adapter + prompt │
│ - Starts trace_id (W3C) │
└──────┬─────────────────────┬────────────────────┘
│ │
(pull prompt) │ │ (pull config)
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ Langfuse Prompt │ │ Model registry │
│ Registry (v12) │ │ (adapter v7) │
└─────────┬──────────┘ └──────────┬───────────┘
│ │
└──────────┬───────────────┘
│ payload ready
▼
┌──────────────────────────────────────────┐
│ vLLM engine (K8s Operator) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pool prefill │ → │ Pool decode │ │
│ │ (H100×N) │ │ (H100×M) │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ KV cache fabric │ │
│ └──────────────────┘ │
│ - prefix caching of the system prompt │
│ - PagedAttention │
└──────┬───────────────────────────────────┘
│ tokens stream
▼
┌─────────────────────────────────────────┐
│ User sees the answer + thumbs/UX UI │
└─────────────────┬───────────────────────┘
│ feedback (15 s later)
▼
┌─────────────────────────────────────────┐
│ feedback_signals (Postgres) │
│ + Langfuse scores │
└─────────────────┬───────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
triage quarterly Retrain cycle dataset_id
root cause or incident-driven enriched (DVC)
│
▼
Tune of v8
(next cycle)
In parallel throughout the request, OTel instrumentation:
spans → Tempo / Jaeger ; events → Langfuse ; metrics → Prometheus

The on-premise stack applied

Taking the above to a generic on-premise consultant-profile infrastructure (RTX 4090 + 4×H100 SXM cluster):

Layer	Typical resources
Network plane	Edge LB (HAProxy / nginx ingress) + Cilium CNI with BGP, covered in Cilium BGP
K8s compute plane	RKE2 with two manager nodes + a GPU node pool
Production GPU plane	4× H100 SXM (NVLink, 80 GB each), partitioned via MIG into prefill/decode pools
Development GPU plane	1× RTX 4090 (24 GB) for offline evals, drift-check embeddings, smoke tests
Storage plane	MinIO or Ceph object store; DVC remote + lakeFS backend
OLTP data plane	Postgres 18 with replication; pgvector 0.8 for small cases
Vector plane	Qdrant or Milvus for large RAG corpora
Stream plane	Kafka (Redpanda / pure Apache) + Schema Registry; CDC with Debezium or Flink CDC
Observability plane	OTel Collector + Tempo (traces) + Prometheus (metrics) + Loki (logs); Langfuse for the LLM-specific part
Runtime security plane	Tetragon, covered in the runtime security post

The real density is not the sum of the boxes: it is the operations that tie the boxes together. A cluster with every piece but no versioning discipline, no end-to-end trace_id propagation, no schema contracts and no cadenced retraining is a cluster that serves LLM once and then ages. The difference between a project and a platform is exactly that.

Ten bridges between stages where the system breaks

The route reveals something important: failures are rarely inside a stage; they are on the bridges between stages. Ten common bridges:

Data → Tune: the dataset does not propagate its (dataset_id, dataset_version) to the trainer. The same dataset trained twice produces two model_id values that cannot be told apart.
Tune → Eval: the trained model does not propagate its lineage to the eval run. The eval passes, but there is no record of which dataset it was trained on. Three months later, irreproducible.
Eval → Deploy: promotion happens without the serving system recording which version of the adapter it is serving at each moment. The day the model gives a dangerous answer, nobody knows which adapter answered.
Deploy → Observe: the engine does not emit gen_ai.request.adapter, gen_ai.prompt.version, gen_ai.dataset.version as span attributes. The traces exist but cannot be crossed with the lineage.
Observe → Retrain: the feedback is captured in a tool (Langfuse, Phoenix) but nobody reads it. The Retrain stage “exists”, but feedback piles up untriaged.
Retrain → Data: the enriched dataset goes into the next Tune without passing through versioning discipline, schema contract and holdout check. Silent contamination of the golden set.
Prompt versioning ↔ everything: the prompt_id, prompt_version does not propagate to the spans. The day the team discovers that a prompt change regressed the system, it cannot isolate which one or when.
Data versioning ↔ everything: the dataset_id, dataset_version does not show up in experiment tracking. “v8 is retrained” but nobody can prove it was over the enriched dataset and not the old one.
MCP ↔ tools: the system invokes tools (cancellation, policy changes) but does not record a gen_ai.tool.invocation_id linked to the trace. The actions end up dissociated from the answer that generated them.
Schema Registry ↔ data: datasets version content but not schema. A breaking change in expected_output breaks the eval silently; nobody notices anything until a human reviews the results.

The bridges are covered throughout the blog. Operations enforce them. The team’s culture keeps them.

How to walk through the blog

If you arrive at this post from outside and want a reading route:

The map: The six-stage LLMOps pipeline — the master map of everything else.
The context: MLOps specific to LLMs in 2026 — the landscape and why LLMOps is not classic MLOps.
Inference from the inside out: KV cache → PagedAttention deep dive → Disaggregated serving → Multi-tenant GPU cluster → vLLM on K8s → LLM operators on K8s.
Data: Data versioning with DVC and lakeFS → PostgreSQL + Qdrant ingestion → RAG over Kafka.
Tune: Continuous fine-tuning in production.
Eval: Evals: the layer after tracing → Guardrails and safety.
Observe: AgentSight LLM tracing → MCP observability with OTel → On-device eBPF + drift.
Retrain: Closing the feedback → dataset → adapter loop.
Cross-cutting: Prompt versioning with Langfuse and MLflow.
Supporting infrastructure (the base everything is built on): RKE2 with Cilium BGP, Hubble + eBPF observability, Tetragon runtime security.

What we have not covered (yet)

At the first level the main things are there. The following posts on the blog, when the topics justify it, could go deeper into:

Schema Registry for LLM data and prompts: the other half of the data contract.
A dedicated AI Gateway: LiteLLM, Portkey, Kong AI Gateway as a control plane.
OTel gen_ai semantic conventions: the emerging standard that ties the seven identifiers of the “identity” block into well-formed spans.
Federated learning over regulated customer data: how to train without centralising the corpus.
Capacity planning for shared multi-tenant clusters.
Disaster recovery of an LLM service: how to reproduce the state of the system 30 days back.
Cost accounting per tenant: tokens × weights × adapter × infrastructure → invoice.

References

W3C Trace Context — end-to-end propagation of traceparent and tracestate.
OpenTelemetry GenAI Semantic Conventions — gen_ai.* attributes for LLM spans.
Langfuse documentation — observability and prompt registry.
vLLM documentation — production inference engine with PagedAttention and LoRA hot-swap.
Kubernetes Operators — the declarative management pattern.
MLflow Tracking and Model Registry — lineage of runs and input artifacts.
DVC and lakeFS — dataset versioning, unified in Nov 2025.
OpenLineage — an open standard for lineage events between systems.
ENS / NIS2: compliance frameworks that apply to operators in the EU; recommended reading for the context in which the scenario operates.

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

Tue, 16 Jun 2026 12:00:00 +0200

Notation: amounts in euros (N €), decimals with a point, thousands with a thin space. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The “data” series has produced three independent measurement axes: cost per million tokens (OpenCost + LiteLLM), performance under an SLO (GuideLLM + AIPerf) and energy per token (DCGM + Kepler). The problem is that the three have been measured in different articles, with different loads and at different times: they are not comparable with each other. This closing article describes the integrated harness that runs the three axes in the same experiment, on the same node (4×H100 SXM, a generic reference), with all the metadata fixed, the output in versioned JSON/CSV and an idempotent Kubernetes Job. The result is the 3-axis scorecard (€/1M tok, Wh/token, TTFT/ITL P99) that allows configurations to be compared on a multi-objective Pareto frontier and any figure to be audited with the bench to reproduce it.

Why the three axes must be measured together

The series’ opening post (The three axes) established the identity:

$$\text{CPM} = \frac{\text{cost/h}}{\text{throughput (tok/s)} \times 3.6 \times 10^{-3}}$$ $$\text{energy/token (Wh)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)} \times 3\,600}$$

Throughput is the common denominator. If it is measured in different experiments, different time, different load, different GPU temperature, then CPM and energy/token do not share a denominator: they are three anecdotes, not a scorecard. The harness captures them in the same time window, over the same load, with Prometheus windows aligned to the second. Only then is the scorecard row coherent by construction.

The second reason is reproducibility. The post Bias and reproducibility in benchmarking listed twelve biases that invalidate comparisons: an unpinned engine, an undeclared tokeniser, unfixed input/output lengths, missing warmup, a client outside the cluster. The harness removes all of them because the metadata are part of the Job, not of the documentation.

Architecture of the integrated bench

The harness has four layers. Each one is OSS, exports to Prometheus and lives in the same Kubernetes namespace:

Layer	Tool(s)	Primary metric	Export protocol
Performance	GuideLLM (SLO sweep) + AIPerf	TTFT P99, ITL P99, goodput (tok/s)	native JSON/CSV + `/metrics` OpenMetrics
Cost	OpenCost + LiteLLM proxy	CPM (€/1M tok), cost/request	REST API + Prometheus scrape
Energy (GPU)	DCGM Exporter	`DCGM_FI_DEV_POWER_USAGE` (W), `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` (mJ)	Prometheus DaemonSet
Energy (pod)	Kepler	`kepler_container_joules_total`, energy per pod	Prometheus DaemonSet

MLPerf Power is used as an external comparability reference: its published results, with hardware documented in detail, allow the harness figures to be calibrated for plausibility. The in-house bench does not aim to be an MLPerf submission, but to be reproducible on your own cluster.

The Kubernetes Job: the full YAML

The experiment runs as a versioned Kubernetes Job. All the relevant metadata are environment variables declared in the manifest: not in ad-hoc scripts, not in external documentation. The Job is idempotent (same name = same run) and leaves traces in the pod log and in the output volume.

apiVersion: batch/v1
kind: Job
metadata:
 name: bench-llama3-70b-fp8-h100x4-20260616
 namespace: benchmark
 labels:
 bench/model: llama3-70b
 bench/precision: fp8
 bench/engine: vllm-0.9.1
 bench/hardware: h100x4-sxm
 bench/isl: "1024"
 bench/osl: "256"
 bench/concurrency: "32"
 bench/tokenizer: meta-llama-3-tokenizer-v3
 bench/run-id: "20260616T1200"
spec:
 backoffLimit: 0
 template:
 spec:
 restartPolicy: Never
 serviceAccountName: bench-runner
 volumes:
 - name: results
 persistentVolumeClaim:
 claimName: bench-results-pvc
 initContainers:
 # Warmup: 60 s of traffic before the experiment
 - name: warmup
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - concurrent
 - --rate
 - "4"
 - --max-seconds
 - "60"
 - --data
 - prompt_tokens=1024,output_tokens=256
 env:
 - name: GUIDELLM_ENV
 value: production
 containers:
 - name: guidellm-sweep
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - sweep
 - --max-seconds
 - "120"
 - --data
 - prompt_tokens=1024,output_tokens=256
 - --output-path
 - /results/guidellm-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

 - name: aiperf-concurrent
 image: nvcr.io/nvidia/aiperf:0.2.0
 command:
 - aiperf
 - profile
 - --url
 - http://vllm-svc.inference.svc.cluster.local:8000/v1
 - --model
 - meta-llama/Meta-Llama-3-70B-Instruct
 - --concurrency
 - "4,8,16,32"
 - --input-tokens
 - "1024"
 - --output-tokens
 - "256"
 - --num-requests
 - "200"
 - --output
 - /results/aiperf-$(BENCH_RUN_ID).json
 volumeMounts:
 - name: results
 mountPath: /results

 - name: scorecard-exporter
 image: python:3.12-slim
 command:
 - python
 - /scripts/export_scorecard.py
 - --run-id
 - "$(BENCH_RUN_ID)"
 - --prometheus
 - http://prometheus.monitoring.svc.cluster.local:9090
 - --output
 - /results/scorecard-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

The Job name (bench-llama3-70b-fp8-h100x4-20260616) is the unique identifier of the run. Changing it is enough to register a variant. The labels are the metadata that the scorecard-exporter reads to enrich the output JSON.

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

GuideLLM (a vLLM project) generates realistic traffic patterns, synchronous, concurrent, poisson, throughput, sweep, and captures full TTFT and ITL distributions (Red Hat Developer). The sweep mode ramps from idle to saturation over 10 rounds and identifies the knee: the maximum load where goodput ≈ throughput under the declared SLO. The output is JSON/CSV with all the percentiles per round, ready to be versioned.

For the harness, the bench’s reference SLO is:

Metric	Threshold	Percentile
TTFT	500 ms	P99
ITL (TPOT)	50 ms/tok	P95
Error rate	0.5 %	—

The Job command already appears in the YAML above. The JSON output includes, per round: rate (req/s), TTFT (P50/P95/P99), ITL (P50/P95/P99), throughput (tok/s) and goodput (tok/s). The goodput under the SLO at the knee is the value that enters the scorecard.

For the cross-verification layer of the data and comparability with published results, see the post GuideLLM in depth.

AIPerf — NVIDIA’s concurrency profiler

AIPerf (the successor to GenAI-Perf, repository ai-dynamo/aiperf) measures TTFT, ITL, throughput and latency distributions at fixed concurrencies (GitHub ai-dynamo/aiperf). Where GuideLLM gives the automatic sweep up to the knee, AIPerf gives the detailed profile at specific concurrencies (4, 8, 16, 32 in the example): it allows the throughput-latency curve to be characterised point by point.

The two are complementary: GuideLLM finds the knee automatically; AIPerf confirms it and characterises the behaviour in the neighbourhood of that knee. Both outputs go to the results volume with the same BENCH_RUN_ID. The deep analysis of AIPerf/GenAI-Perf is in GenAI-Perf in depth.

Metrics the harness extracts from AIPerf for the scorecard:

# TTFT P99 at concurrency 16 (the closest to the sweep's knee):
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .ttft_ms.p99'

# Goodput (tok/s) at concurrency 16:
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .output_token_throughput'

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

OpenCost (CNCF incubating, opencost.io) allocates Kubernetes cost to namespace, label, pod and container in real time (GitHub opencost/opencost). For the harness, OpenCost answers the question: how much does the vllm-svc pod cost in euros per hour during the experiment’s window?

The harness calls the OpenCost REST API when the experiment finishes:

# Cost of the inference namespace over the experiment window (1 hour)
curl -s "http://opencost.monitoring.svc.cluster.local:9003/allocation" \
 --data-urlencode 'window=2026-06-16T12:00:00Z,2026-06-16T13:00:00Z' \
 --data-urlencode 'aggregate=namespace' \
 --data-urlencode 'namespace=inference' \
 | jq '.data[0].inference.totalCost'

The returned cost (in EUR, configured with the node’s real prices) is divided by the throughput measured in that same window to obtain the CPM:

$$\text{CPM} = \frac{\text{cost}_\text{namespace/h} \times 10^6}{\text{goodput (tok/s)} \times 3\,600}$$

The post OpenCost: cost allocation in Kubernetes details the price configuration and the allocation by GPU label.

LiteLLM — the per-request token counter

LiteLLM (litellm.ai, GitHub BerriAI/litellm) acts as an OpenAI-compatible proxy with token accounting per request, model and team. In the harness, GuideLLM and AIPerf point at the LiteLLM endpoint (which in turn forwards to vLLM): every request is recorded with prompt_tokens, completion_tokens and cost (using the custom pricing configured for the on-prem node).

# litellm-config.yaml (fragment of custom on-prem pricing)
model_list:
 - model_name: llama3-70b-fp8
 litellm_params:
 model: openai/meta-llama/Meta-Llama-3-70B-Instruct
 api_base: http://vllm-svc.inference.svc.cluster.local:8000/v1
 api_key: sk-dummy
 input_cost_per_token: 0.00000109 # 1.09 EUR/1M tok on-prem
 output_cost_per_token: 0.00000109

The LiteLLM records are exported to Prometheus as litellm_request_total_tokens and litellm_spend_metric_total, which the scorecard exporter consumes to compute the real CPM per token type (input vs output).

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

DCGM Exporter (GitHub NVIDIA/dcgm-exporter, docs) exposes GPU metrics on /metrics for Prometheus as a DaemonSet on the GPU nodes. The two energy metrics the harness uses:

DCGM metric	Type	Unit	Use in the harness
`DCGM_FI_DEV_POWER_USAGE`	gauge	W	instantaneous power per GPU
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	counter	mJ	energy accumulated since boot

To compute the energy consumed during the experiment’s window (without the idle baseline), the harness takes the difference of the counter before and after the sweep:

# Deployment as a DaemonSet (fragment of the official chart)
helm repo add gpu-helm-charts \
 https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
 --namespace monitoring \
 --set serviceMonitor.enabled=true \
 --set serviceMonitor.interval=15s

The DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION field (in mJ) allows the experiment’s energy to be computed with hardware-counter accuracy, not with an estimate:

$$\text{experiment energy (J)} = \left(\text{counter}_\text{end} - \text{counter}_\text{start}\right) \times 10^{-3}$$ $$\text{Wh/token} = \frac{\text{experiment energy (J)}}{3\,600 \times \text{generated tokens}}$$

Kepler — pod-level energy

Kepler (CNCF sandbox, GitHub sustainable-computing-io/kepler) uses eBPF to estimate energy consumption at container and pod level, exporting kepler_container_joules_total to Prometheus (Red Hat Emerging Technologies). It combines RAPL (CPU/DRAM), NVML (GPU) and regression models when no sensors are available.

In the harness, Kepler complements DCGM: DCGM gives the hardware measurement of the GPU (more precise for GPU-intensive loads such as LLM inference), while Kepler attributes the energy to the specific vLLM pod (including the node’s CPU contribution). The main metric the harness consumes:

# Energy of the vllm-svc pod during the experiment (J)
increase(
 kepler_container_joules_total{
 container_namespace="inference",
 container_name="vllm"
 }[${BENCH_DURATION}]
)

The comparative analysis of DCGM vs Kepler vs Zeus is in Energy tools: deployment, precision and overhead.

The MLPerf Power reference

MLPerf Power (MLCommons, IEEE HPCA 2025 paper) establishes the protocol for measuring the energy efficiency of ML systems with high-precision external measurement. The in-house bench is not an MLPerf submission (that requires external power meters and committee review), but its published results are the calibration reference: if the harness gives a result of the same order as the MLPerf submission for the same hardware with the same model, the measurement is plausible. If it differs by more than a factor of 2, there is a methodological problem. See the post MLPerf Power: energy efficiency for the current reference data.

Unifying the metrics: the PromQL queries

The scorecard-exporter is the Job container that, once GuideLLM and AIPerf have finished, collects all the metrics from Prometheus and builds the JSON of the scorecard row. The key queries:

# Mean GPU power during the sweep (W) — 4×H100 node
avg_over_time(
 sum(DCGM_FI_DEV_POWER_USAGE{Hostname=~"gpu-node-.*"})[${BENCH_DURATION}:15s]
)

# Total GPU energy during the sweep (mJ → convert to J)
(
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset 0
 -
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset ${BENCH_DURATION}
) * 0.001

# Tokens generated during the sweep (from LiteLLM)
increase(
 litellm_request_total_tokens{model="llama3-70b-fp8", token_type="completion"}[${BENCH_DURATION}]
)

# Cost of the inference namespace over the window (from the OpenCost API)
# → REST call at the start and at the end of the Job, difference of accrual

The ${BENCH_DURATION} variable is the real duration of the sweep (in Prometheus format, e.g. 22m), which the exporter computes as end_ts - start_ts and substitutes into every query.

The output JSON of each run has this structure:

{
 "run_id": "20260616T1200",
 "metadata": {
 "model": "meta-llama/Meta-Llama-3-70B-Instruct",
 "engine": "vllm-0.9.1",
 "precision": "fp8",
 "hardware": "4xH100-SXM-80GB",
 "isl_tokens": 1024,
 "osl_tokens": 256,
 "tokenizer": "meta-llama-3-tokenizer-v3",
 "slo_ttft_p99_ms": 500,
 "slo_itl_p95_ms": 50,
 "bench_tool_guidellm": "0.4.2",
 "bench_tool_aiperf": "0.2.0",
 "dcgm_exporter": "3.3.9",
 "kepler": "0.10.2"
 },
 "performance": {
 "goodput_tok_s": 3120,
 "ttft_p99_ms": 487,
 "itl_p95_ms": 42,
 "throughput_peak_tok_s": 3890,
 "elbow_concurrency": 28
 },
 "cost": {
 "cpm_eur_1m": 0.97,
 "gpu_cost_eur_h": 10.8,
 "cost_window_eur": 3.24
 },
 "energy": {
 "wh_per_token": 0.00044,
 "j_per_token": 1.58,
 "power_mean_w": 4924,
 "energy_total_kwh": 0.287,
 "pue": 1.4,
 "wh_per_token_pue_adjusted": 0.000616
 },
 "carbon": {
 "grid_intensity_gco2_kwh": 40,
 "co2_per_1m_tokens_g": 24.6
 }
}

All the fields are computed from the same time windows. The JSON is versioned in git alongside the harness code. Reproducing any scorecard row is: git checkout <run-id> && kubectl apply -f job.yaml.

The 3-axis scorecard: table and interpretation

The following table illustrates how five configurations of the same model (Llama 3 70B) on the same reference node (4×H100 SXM) compare with the harness. The figures are illustrative of the order of magnitude (the bench fills them in with real measurements):

Config	Engine	Precision	CPM (€/1M)	Goodput (tok/s)	Wh/tok	TTFT P99 (ms)	ITL P95 (ms)	CO2 /1M (g, FR)
A	vLLM 0.9	FP16	1.64	1,890	0.00074	498	49	29.6
B	vLLM 0.9	FP8	0.97	3,120	0.00044	487	42	17.6
C	SGLang 0.4	FP8	0.88	3,410	0.00041	421	38	16.4
D	vLLM 0.9	FP8, ISL 512	1.31	2,240	0.00056	294	31	22.4
E	vLLM 0.9	FP8, max-batch 128	0.84	3,590	0.00040	721	55	16.0

How to read the table:

B vs A: FP8 over FP16 cuts CPM by 41 % and energy per token by 41 % thanks to the same higher throughput. Both meet the SLO (TTFT P99 < 500 ms, ITL P95 < 50 ms). B dominates A on all three axes: it is Pareto-superior.
C vs B: SGLang gives 9 % more goodput and 9 % lower CPM with FP8. Both meet the SLO. C is Pareto-superior to B (if the engine is indifferent for the stack).
D vs B: a shorter ISL (512 tok) lowers latency (TTFT P99 294 ms vs 487 ms) but lowers goodput and raises CPM. If the use case demands TTFT < 300 ms, D is the candidate; if TTFT < 500 ms is enough, B is better on cost and energy.
E vs C: max-batch 128 raises goodput (+5 %) and lowers CPM, but the TTFT P99 breaks the SLO (721 ms > 500 ms) and so does the ITL P95 (55 ms > 50 ms). E has the best raw throughput but is outside the SLO: it is not a valid candidate for the interactive chat use case.

The scorecard’s Pareto frontier, under the declared SLO, includes only A, B, C and D (E falls out for breaking the SLO). Of those four, C dominates B which dominates A. D only enters the frontier if the use case demands TTFT P99 < 300 ms. The decision is not a number; it is the whole row plus the SLO.

The multi-objective Pareto frontier

With three minimisation metrics, CPM (€/1M tok), Wh/tok and TTFT P99 (ms), the Pareto frontier is defined as the set of configurations where none dominates another on all three axes simultaneously. Formally, config $ i $ dominates config $ j $ if:

$$\text{CPM}_i \leq \text{CPM}_j \;\land\; \text{Wh/tok}_i \leq \text{Wh/tok}_j \;\land\; \text{TTFT}_{i} \leq \text{TTFT}_{j}$$

with at least one strict inequality. The scorecard allows it to be computed over the set of runs with a simple vector operation. In the table’s example, the frontier under the SLO (with E excluded) is $\{C, D\}$: C dominates on cost/energy/goodput; D dominates on latency. Between C and D the choice depends on the TTFT SLO of the specific use case.

Reproducible design: the twelve mandatory metadata

The checklist that closes the bias dossier (Bias and reproducibility). For the harness JSON to be auditable, the twelve fields must be present in every run:

#	Field	Example	Why it is mandatory
1	Serving engine version	`vllm-0.9.1`	the same model can perform differently across versions
2	Model version (commit/tag)	`meta-llama/Meta-Llama-3-70B-Instruct@sha256:abc`	avoids ambiguity between variants of the same name
3	Precision	`fp8`	FP16 vs FP8 changes throughput and latency
4	Tokeniser and version	`meta-llama-3-tokenizer-v3`	the ISL/OSL in tokens depends on the tokeniser
5	ISL (input sequence length)	`1024` tok	changes the prefill time and the knee
6	OSL (output sequence length)	`256` tok	changes the decode time
7	Hardware (model and count)	`4×H100-SXM-80GB`	the basis of any comparison
8	Maximum concurrency tested	`32`	defines the sweep’s range
9	Bench tool version	`guidellm-0.4.2`, `aiperf-0.2.0`	different versions can give different metrics
10	Declared SLO	`TTFT P99 < 500 ms, ITL P95 < 50 ms`	the knee depends on the SLO; without it there is no goodput
11	Warmup duration	`60 s`	without warmup the KV cache is cold and the first rounds are biased
12	Harness version / run-id	`20260616T1200`	identifies the run so it can be reproduced with `git checkout`

A scorecard missing any of these twelve fields is an anecdote. With all twelve, it is an auditable datum. The run-id in the Job name (bench-llama3-70b-fp8-h100x4-20260616) implicitly incorporates fields 1-3 and 7, forcing the Job name to change with every variant, which makes it impossible to overlap runs in the same namespace.

Idempotence and versioning of results

The Kubernetes Job is idempotent: if the same manifest is applied again (same name), the Job already exists and Kubernetes does not relaunch it (backoffLimit: 0 policy). This avoids accidental duplicate runs. To relaunch, the Job has to be deleted (kubectl delete job <name>) or the run-id changed.

Results are versioned with the following directory convention in the harness’s git repository:

results/
20260616T1200/
guidellm-20260616T1200.json
aiperf-20260616T1200.json
scorecard-20260616T1200.json
20260617T0900/
guidellm-20260617T0900.json
aiperf-20260617T0900.json
scorecard-20260617T0900.json
scorecard-aggregate.csv # all rows, for analysis and charts

The scorecard-aggregate.csv is the scorecard table in flat format: each row is a run, each column a field of the JSON. It is generated with:

# Generates the aggregate CSV from all the JSON files in results/
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

The CSV is what feeds the Pareto charts, the reports and the architecture decisions. It is versioned in git; its diff is the configuration change.

How it connects with the other posts in the series

The harness is not a standalone article: it is the synthesis of the twenty-eight articles. Each tool has its deep dive:

Performance: GuideLLM in depth — sweep, SLO, goodput, knee; GenAI-Perf / AIPerf in depth — concurrency, rate sweep, distribution metrics.
Cost: OpenCost: cost allocation in Kubernetes — allocation by namespace/label, custom on-prem prices.
Energy: Energy tools: deployment, precision and overhead — DCGM vs Kepler vs Zeus, instrumentation overhead, relative precision; MLPerf Power: energy efficiency — the external calibration reference.
Theoretical framework: The three axes: cost, performance and energy — the CPM/energy/throughput identity that justifies measuring them together.
Bias: Bias and reproducibility in benchmarking — the twelve biases the harness removes.

The harness closes the series’ dossier because it makes the opening article’s promise possible: “when someone disputes a number in the proposal, the answer is not ‘a blog says so’, but ’this is the bench, this is the methodology, reproduce it’”. With the run-id and the git repo, reproducing is one command.

Full flow of a benchmarking session

The harness’s standard operating procedure, from start to finish:

# 1. Make sure the inference endpoint is up
kubectl get svc vllm-svc -n inference

# 2. Verify that DCGM and Kepler are scraping
curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
 --data-urlencode 'query=DCGM_FI_DEV_POWER_USAGE' | jq '.data.result | length'

# 3. Record the energy counter BEFORE the experiment
PRE_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 4. Launch the Job (unique name per run)
kubectl apply -f jobs/bench-llama3-70b-fp8-h100x4-20260616.yaml

# 5. Wait for it to finish
kubectl wait --for=condition=complete job/bench-llama3-70b-fp8-h100x4-20260616 \
 -n benchmark --timeout=3600s

# 6. Record the energy counter AFTER
POST_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 7. Collect the results from the volume
kubectl cp benchmark/bench-run-pod:/results/ ./results/20260616T1200/

# 8. Compute the experiment's energy (mJ → kWh)
python scripts/calc_energy.py \
 --pre $PRE_ENERGY --post $POST_ENERGY \
 --run-id 20260616T1200

# 9. Add a row to the aggregate CSV
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

# 10. Version it
git add results/20260616T1200/ scorecard-aggregate.csv
git commit -m "bench: llama3-70b fp8 4xH100 ISL1024 OSL256 (20260616T1200)"

Steps 6-9 are candidates for automation as a second post-processing Job launched when the main one completes (with ownerReferences or a clean-up CronJob). In the harness’s basic state, the manual steps are precisely the ones that force a review of the result before versioning it.

Cost of the experiment

A GuideLLM sweep of 10 rounds at 120 seconds per round occupies the node for ~22-26 minutes. The AIPerf profile at 4 concurrencies (200 requests each) adds ~8-12 minutes. Total per run: ~35-40 minutes of a 4×H100 node.

At the reference amortised cost (~10.8 €/h for 4×H100 on-prem):

$$\text{cost per run} \approx 10.8 \times \frac{38}{60} \approx 6.84 \text{ EUR}$$

At the reference European cloud price (4 × 2.73 = 10.92 €/h):

$$\text{cost per run (cloud)} \approx 10.92 \times \frac{38}{60} \approx 6.92 \text{ EUR}$$

A continuous benchmarking programme (10 configurations × 3 models × 2 sweeps/week) adds up to ~415 EUR/week. That is why the harness includes short sweeps (5 rounds, ~15 minutes) for CI and full sweeps for releases.

What the harness does not measure (and why)

Missing dimension	Reason	Where it is covered
Output quality (MMLU, HumanEval)	requires a quality evaluation bench, not a serving one	LLM quality benchmarks
Training throughput	the harness is exclusively for inference	outside the scope of the “data” series
Network and storage cost	OpenCost can attribute it, but it requires additional configuration	OpenCost: cost allocation
Hourly carbon intensity	use the ElectricityMaps API + the measured energy	From the watt to carbon: PUE and grid mix
Client-server network latency	the Job runs inside the cluster; WAN latency requires an external client	document as additional metadata

Sources

GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
GuideLLM · PyPI — https://pypi.org/project/guidellm/
Red Hat Developer · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
AIPerf · GitHub (ai-dynamo/aiperf) — https://github.com/ai-dynamo/aiperf
GenAI-Perf · documentación NVIDIA Triton — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · LLM performance benchmarking con GenAI-Perf — https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf
NVIDIA NIM Benchmarking · métricas (TTFT, ITL, throughput) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
OpenCost · web oficial — https://opencost.io/
OpenCost · GitHub (opencost/opencost) — https://github.com/opencost/opencost
OpenCost · CNCF blog (sandbox → incubating) — https://www.cncf.io/blog/2022/12/06/opencost-a-new-cncf-sandbox-project-for-real-time-kubernetes-cost-monitoring/
LiteLLM · web oficial — https://www.litellm.ai/
LiteLLM · GitHub (BerriAI/litellm) — https://github.com/BerriAI/litellm
LiteLLM · Spend Tracking docs — https://docs.litellm.ai/docs/proxy/cost_tracking
NVIDIA DCGM Exporter · documentación oficial — https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
NVIDIA DCGM Exporter · GitHub — https://github.com/NVIDIA/dcgm-exporter
NVIDIA DCGM Exporter · métricas CSV — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/dcp-metrics-included.csv
Kepler (Kubernetes-based Efficient Power Level Exporter) · GitHub — https://github.com/sustainable-computing-io/kepler
Red Hat Emerging Technologies · Introducing Kepler — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
MLCommons · MLPerf Inference Datacenter benchmark — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Power presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
IEEE Xplore · MLPerf Power paper (μWatts to MWatts) — https://ieeexplore.ieee.org/document/10946778/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · Efficient and reproducible LLM inference: MLPerf Inference v5.1 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
BentoML · Beyond tokens-per-second: cost, speed and quality in LLM inference — https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

Tue, 16 Jun 2026 11:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator (1{,}234). Generic example hardware: 4×H100 SXM5 80 GB nodes. The dollar sign is not used (it is the formula delimiter).

TL;DR

With an SLO of TTFT P99 ≤ 300 ms and ITL P99 ≤ 50 ms for a production chat, peak GPU utilisation must stay at ≤ 63 %.
A service receiving 5M requests/day with 512 input tokens and 256 output tokens on average generates an hourly peak of ~11,600 tok/s (with a peak factor of 1.8×). With vLLM on H100 SXM5 (Llama-3.3 70B FP8, ~1,850 tok/s at 50 concurrent requests), ≥ 10 GPUs are needed at peak to meet the SLO, which translates into 3 4×H100 nodes (with headroom).
The all-in cost of those 3 nodes is ~83,300 € per year per node (3-year capex + base opex); the €/1M tokens in the base scenario at 63 % utilisation is ~0.37 €.
Against AWS p5 on-demand (~3.90 USD/GPU-hour after the 44 % cut of Jun. 2025), the utilisation break-even sits at ~55 %; against neocloud on-demand (~2.01 USD/GPU-hour), there is no feasible break-even at medium utilisation.
Capex payback happens between 13 and 24 months if sustained utilisation exceeds 70 %.

1. The sizing chain: from demand to the number of GPUs

Sizing an inference platform follows a causal chain of five steps. Each step has a formula; none can be skipped.

1.1 Step 1 — Characterise the demand

Parameter	Symbol	Example
Requests per day (average)	$D$	5,000,000
Input tokens per request (average)	$L_{\text{in}}$	512
Output tokens per request (average)	$L_{\text{out}}$	256
Peak factor (peak hour vs average ratio)	$k_{\text{peak}}$	1.8
Hours per day with significant traffic	$H$	16

The average output throughput:

$$\dot{T}_{\text{avg}} = \frac{D \times L_{\text{out}}}{86{,}400\,\text{s}} = \frac{5{,}000{,}000 \times 256}{86{,}400} \approx 14{,}815\;\text{tok/s}$$

The peak throughput (busy hour):

$$\dot{T}_{\text{peak}} = k_{\text{peak}} \times \frac{D \times L_{\text{out}}}{H \times 3{,}600} = 1.8 \times \frac{5{,}000{,}000 \times 256}{16 \times 3{,}600} \approx 40{,}000\;\text{tok/s}$$

Note: if the traffic profile has very pronounced peaks (peak/average ratio > 3), sizing is done for the peak and average utilisation falls; the sensitivity analysis in §5 quantifies the effect.

1.2 Step 2 — Set the SLO and derive the maximum utilisation

The latency SLO imposes a ceiling on GPU utilisation. Using queueing theory (M/G/1 model):

$$\rho_{\max} \approx 1 - \frac{1}{\sqrt{1 + C_{s}^{2}}} \cdot \frac{W_{\text{queue}}^{*}}{\bar{s}}$$

where $\rho$ is the utilisation, $W_{\text{queue}}^{*}$ the maximum admissible queueing time and $\bar{s}$ the average service time per request. For the simplified M/M/1 case with prefill time dominant:

$$\rho_{\max} = 1 - \frac{W_{\text{queue}}^{*}}{\bar{s} \cdot (1 + W_{\text{queue}}^{*}/\bar{s})}$$

The practical reference table (derived from the Spheron/Little’s Law formula, 2026):

SLO TTFT P99	Maximum utilisation $\rho_{\max}$
200 ms	55 %
300 ms	63 %
400 ms	70 %
500 ms	75 %

For our example (SLO 300 ms P99): $\rho_{\max} = 0.63$.

The ITL SLO imposes an additional constraint: the inference engine must be able to generate the next token in ≤ 50 ms. On H100 SXM5 with vLLM and Llama-3.3 70B FP8, the ITL P50 at 50 concurrent requests is ~20 ms, with P99 ~45 ms. ITL is the dominant constraint only when the batch size is very high (>64 sequences) or VRAM is nearly full.

1.3 Step 3 — Required throughput and throughput per GPU

The throughput the cluster must serve at peak, while respecting $\rho_{\max}$:

$$\dot{T}_{\text{required}} = \frac{\dot{T}_{\text{peak}}}{\rho_{\max}} = \frac{40{,}000}{0.63} \approx 63{,}500\;\text{tok/s (installed capacity)}$$

The throughput per GPU (reference benchmark, vLLM v0.18.0, Llama-3.3 70B FP8, H100 SXM5 80 GB, 50 concurrent requests):

Engine	Throughput (tok/s per GPU)	TTFT P50 / P95 (50 req)	Source
vLLM 0.18.0	1,850	380 ms / 720 ms	Spheron benchmarks, Mar. 2026
SGLang 0.5.9	1,920	360 ms / 680 ms	Spheron benchmarks, Mar. 2026
TensorRT-LLM 1.2.0	2,100	340 ms / 620 ms	Spheron benchmarks, Mar. 2026

The throughput data correspond to the test with Llama 3.3 70B Instruct FP8, 512 input / 256 output tokens, 50 concurrent requests, on bare-metal H100 SXM5. See the engine analysis in serving engines Pareto comparison.

We use vLLM as the general-purpose production reference: $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s.

1.4 Step 4 — Number of GPUs and number of nodes

$$N_{\text{GPU}} = \left\lceil \frac{\dot{T}_{\text{required}}}{\dot{T}_{\text{GPU}}} \right\rceil = \left\lceil \frac{63{,}500}{1{,}850} \right\rceil = \lceil 34.3 \rceil = 35\;\text{GPUs}$$

With 4×H100 SXM5 nodes:

$$N_{\text{nodes}} = \left\lceil \frac{N_{\text{GPU}}}{4} \right\rceil = \left\lceil \frac{35}{4} \right\rceil = 9\;\text{nodes}$$

We add 15 % headroom for hardware failures (~5 % annual rate in small clusters), upgrades and unforeseen peaks:

$$N_{\text{nodes, final}} = \lceil 9 \times 1.15 \rceil = 11\;\text{nodes} \approx 44\;\text{GPUs}$$

For the TL;DR example (5M requests/day with a 16 h active profile, k 1.8 and an SLO of 300 ms) the number of nodes is 11. The simplified TL;DR case with k=1 and H=24 gives 3 nodes; the difference illustrates the impact of the hourly profile.

1.5 Summary of the chain

2. From sizing to cost: the TCO model

With $N_{\text{nodes}} = 11$ 4×H100 SXM5 nodes, the TCO follows the model detailed in TCO of an on-premise GPU cluster. Here the compact formula is reproduced and applied to the sized cluster.

2.1 Annual cost per node (base scenario)

$$C_{\text{node/year}} = \frac{\text{node capex}}{\text{years}} + \text{node opex/year}$$

Line item	Value (USD / €)	Note
4×H100 node capex (midpoint)	178,500 USD	GPUs + server + network + storage + rack
3-year amortisation	59,500 USD/year ≈ 55,300 €/year	Straight line
Opex/year (base scenario, 8–16 node cluster)	~28,000 €/year	Energy + staff + maintenance + colocation
Total per node per year	~83,300 €/year	—

For 11 nodes: 915,300 €/year of total fixed cost.

2.2 From €/node-year to €/GPU-hour

$$\text{EUR/GPU-hour} = \frac{C_{\text{node/year}}}{4\;\text{GPUs} \times 8{,}760\;\text{h} \times \rho}$$

Utilisation $\rho$	EUR/GPU-hour
40 %	5.93
55 %	4.31
63 % (SLO 300 ms)	3.76
70 %	3.39
80 %	2.97
100 %	2.38

2.3 From €/GPU-hour to €/1M tokens

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^{6}}{\dot{T}_{\text{GPU}} \times 3{,}600}$$

With $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s (vLLM, Llama-3.3 70B FP8):

Utilisation	EUR/GPU-hour	EUR/1M tokens
40 %	5.93	0.891
55 %	4.31	0.647
63 % (SLO 300 ms)	3.76	0.564
70 %	3.39	0.509
80 %	2.97	0.446
100 %	2.38	0.357

The complete cost/token identity as a function of throughput and utilisation is developed in cost per token and per request.

2.4 Comparison with cloud (equivalent €/1M tokens)

To compare, the cloud price is converted to the €/1M tokens equivalent using the same reference throughput ($\dot{T}_{\text{GPU}} = 1{,}850$ tok/s):

$$\text{EUR/1M tokens (cloud)} = \frac{P_{\text{cloud}} \times 10^{6}}{1{,}850 \times 3{,}600}$$

Provider / Mode	GPU-hour price (USD)	EUR/GPU-hour ($1\,\text{USD} \approx 0.93\,\text{EUR}$)	EUR/1M tokens equiv.
AWS p5 on-demand (post Jun. 2025)	3.90	3.63	0.545
AWS p5 1-year reserved	~2.50	2.33	0.350
CoreWeave on-demand	~2.01	1.87	0.281
CoreWeave 3-year reserved	~1.49	1.39	0.209
GCP A3 on-demand	~3.67	3.41	0.512
Azure ND H100 v5 on-demand	~6.98	6.49	0.975

Sources: IntuitionLabs (Jun. 2026), CloudZero (Jun. 2026), Spheron GPU pricing (May 2026).

3. Break-even and payback of your own investment

3.1 The utilisation break-even formula

The utilisation break-even $u^{*}$ is the utilisation at which the annual on-prem cost per GPU-hour equals the cloud price:

$$u^{*} = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

where $P_{\text{cloud}}$ is the cloud price in the same currency as the on-prem costs.

With the base scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/node/year):

Cloud reference	Cloud price (EUR/GPU-hour)	$u^{*}$ break-even
Azure on-demand (~6.49 EUR)	6.49	23 %
AWS on-demand post-cut (~3.63 EUR)	3.63	41 %
GCP on-demand (~3.41 EUR)	3.41	44 %
CoreWeave on-demand (~1.87 EUR)	1.87	80 %
CoreWeave 3-yr reserved (~1.39 EUR)	1.39	>100 % (impossible)
AWS 1-yr reserved (~2.33 EUR)	2.33	64 %

3.2 Capex payback

Payback is the time $T_{\text{pay}}$ in which the cumulative saving against cloud equals the initial capex:

$$T_{\text{pay}} = \frac{\text{total cluster capex}}{(\text{cloud cost/year}) - (\text{on-prem opex/year})}$$

where the cloud cost/year is computed at the same sustained utilisation.

For the 11-node cluster (total capex 11 × 178,500 USD ≈ 1,825,000 USD ≈ 1,697,000 €):

Cloud reference	Sustained utilisation	Annual saving vs cloud	Payback
AWS on-demand (3.63 EUR/h)	70 %	(3.63 − 3.39 EUR) × 4 × 8760 × 0.70 × 11 nodes ≈ 63,900 €/year	~27 months
AWS on-demand (3.63 EUR/h)	80 %	(3.63 − 2.97) × 4 × 8760 × 0.80 × 11 ≈ 203,600 €/year	~10 months
GCP on-demand (3.41 EUR/h)	70 %	(3.41 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 5,400 €/year	~315 months (not viable)
Azure on-demand (6.49 EUR/h)	70 %	(6.49 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 830,000 €/year	~2 months

The 13-month payback quoted in studies such as Lenovo TCO 2026 corresponds to ~80 % utilisation against high-priced hyperscalers (Azure/AWS before the cut). With current prices (post June 2025, AWS at 3.90 USD), the window widens.

3.3 Simple payback (capex only vs equivalent cloud)

If on-prem opex is omitted and only capex is compared with the gross saving:

$$T_{\text{pay,simple}} = \frac{\text{capex}}{P_{\text{cloud}} \times 4 \times 8{,}760 \times \rho \times N_{\text{nodes}}}$$

Utilisation	AWS (3.63 EUR)	Azure (6.49 EUR)
50 %	54 months	30 months
70 %	39 months	22 months
80 %	34 months	19 months

4. Decision table: when to buy, rent or go hybrid

The table below is a five-dimension Pareto. There is no implicit order between columns; the reading depends on the organisation’s constraints.

Option	EUR/GPU-hour (util. 70 %)	Initial capex	Required utilisation	Data sovereignty	Peak elasticity
On-prem purchase, util. ≥ 70 %	3.39	high (178k USD/node)	≥ 70 % sustained	total	none
On-prem purchase, util. < 50 %	> 4.75	high	< 50 % → loses vs cloud	total	none
Sovereign EU cloud OD (Scaleway/Nebius EU)	2.00–3.59	none	any	yes (EU)	total
AWS p5 on-demand (post Jun. 2025)	3.63	none	any	no (CLOUD Act)	total
CoreWeave on-demand	1.87	none	any	partial (US)	total
CoreWeave 3-year reserved	1.29–1.39	financial commitment	rigid contract	partial (US)	none
AWS 1-year reserved	~2.17	1-year commitment	contract	no (CLOUD Act)	none
Hybrid on-prem base + EU cloud peak	2.50–3.39 (weighted)	medium	base ≥ 70 %, elastic peak	yes (EU)	elastic peak

Cut-off criteria that come before the table:

GDPR sovereignty: if the data is personal or the system is high-risk under the EU AI Act, CoreWeave/AWS are ruled out before any price comparison.
Minimum volume to amortise capex: below ~2M tokens/day sustained over 3 years, on-prem capex does not amortise against AWS on-demand.
Traffic elasticity: peaks >3× the base favour a hybrid or pure cloud; a stable base favours on-prem.

The cost/sovereignty Pareto frontier for GDPR data leaves three options: on-prem, sovereign EU cloud and hybrid. Sustained utilisation and traffic predictability decide between them. See the cross-analysis of the four axes in sovereign on-premise vs hyperscalers.

5. Sensitivity analysis

5.1 Sizing vs hourly profile and peak factor

The number of GPUs grows linearly with $k_{\text{peak}}$ and inversely with $\rho_{\max}$:

$$N_{\text{GPU}} = \left\lceil \frac{D \times L_{\text{out}} \times k_{\text{peak}}}{H \times 3{,}600 \times \rho_{\max} \times \dot{T}_{\text{GPU}}} \right\rceil$$

Peak factor $k$	SLO 300 ms ($\rho_{\max}=0.63$)	SLO 500 ms ($\rho_{\max}=0.75$)
1.2	8 GPUs (2 nodes)	6 GPUs (2 nodes)
1.8	12 GPUs (3 nodes)	10 GPUs (3 nodes)
2.5	16 GPUs (4 nodes)	13 GPUs (4 nodes)
3.5	22 GPUs (6 nodes)	18 GPUs (5 nodes)

(Example simplified to 5M req/day with H=24 to illustrate the sensitivity to the peak factor)

A peak factor of 3.5× triples the number of nodes compared with k=1.2 at the same SLO. Sizing hardware for $k > 2.5$ leaves GPUs idle more than 70 % of the time; peak cloud is more efficient beyond that threshold.

5.2 Break-even vs sustained utilisation

$$u^{*} = \frac{83{,}300}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

Cloud price (EUR/GPU-hour)	$u^{*}$ break-even	Scenario
6.49 (Azure OD)	23 %	On-prem wins almost always
3.63 (AWS OD)	41 %	On-prem wins if util. > 41 %
3.41 (GCP OD)	44 %	—
2.33 (AWS 1-yr reserved)	64 %	On-prem wins if util. > 64 %
1.87 (CoreWeave OD)	80 %	Hard to reach in production
1.39 (CoreWeave 3-yr reserved)	>100 %	On-prem never closes the gap

5.3 Break-even vs energy price

Energy accounts for 6–11 % of total TCO. Its impact on the break-even is moderate:

Energy price (EUR/kWh)	Energy opex/year per node	EUR/GPU-hour (70 % util.)	$u^{*}$ vs AWS OD
0.034 (solar PPA)	1,604 €	3.22	38 %
0.116 (industrial ES, base)	5,475 €	3.39	41 %
0.200 (high tariff)	9,437 €	3.57	43 %

The difference between the cheapest and the most expensive scenario is only 5 percentage points of break-even. The variable that moves the needle is utilisation, not energy.

5.4 Break-even vs demand growth

If demand grows at an annual rate $g$, the average utilisation of the cluster (sized for year 1) rises over time until it saturates and has to be expanded:

$$\rho(t) = \rho_{0} \times (1 + g)^{t}$$

Annual growth $g$	Time to saturation ($\rho \to 100\,\%$)	Decision
10 %	~11 years	Comfortable purchase
30 %	~4 years	Purchase with a review at 3 years
60 %	~2 years	Hybrid: base + elastic cloud
>100 %	<1 year	Pure cloud until it stabilises

For growth above 30 % per year, a buy-only strategy means oversizing for the future peak or re-buying hardware in short cycles. The hybrid (on-prem base + cloud for growth) minimises the capex at risk.

5.5 Headroom: the cost of the safety margin

The 15 % headroom in $N_{\text{nodes}}$ amounts to having ~1.6 extra nodes on average. Its annual cost is:

$$C_{\text{headroom}} = 0.15 \times 83{,}300\;\text{EUR/node/year} \times N_{\text{nodes,base}} \approx 12{,}500 \times 9 = 112{,}500\;\text{EUR/year}$$

That cost is justified by:

GPU failure rate ~5 % per year (in small clusters, documented by Introl, Apr. 2026)
Replacement time of 2–8 weeks (depending on market availability)
Unforeseen peaks of up to 20 % above the estimate

If the service can degrade gracefully (relaxing the TTFT SLO from 300 ms to 500 ms at extreme peak), headroom can be cut to 10 %, saving ~37,500 EUR/year.

6. Sensitivity map: €/GPU-hour and break-even on two axes

The table below crosses utilisation with the opex scenario, showing the all-in EUR/GPU-hour (base scenario, capex/year 55,300 €):

	Low opex (13,000 €/year)	Base opex (28,000 €/year)	High opex (75,000 €/year)
Util. 40 %	4.74	5.93	9.06
Util. 55 %	3.45	4.31	6.59
Util. 63 %	3.01	3.76	5.75
Util. 70 %	2.71	3.39	5.18
Util. 80 %	2.37	2.97	4.54
Util. 100 %	1.90	2.38	3.63

The crossing point with the cloud price (AWS OD: 3.63 EUR):

Low-opex scenario: break-even at ~38 % utilisation
Base-opex scenario: break-even at ~41 % utilisation
High-opex scenario: break-even at ~53 % utilisation

The biggest lever for lowering the break-even is not hardware capex but opex (staff and colocation above all). See the analysis of utilisation as a FinOps lever in GPU utilisation as FinOps.

7. Integration with the rest of the FinOps chain

The sizing of §1 determines the number of nodes; the TCO of §2 gives the cost per hour; the comparison of §3 gives the break-even. Those three numbers feed directly into the other instruments of the series:

Instrument	Input from this article	Output
GuideLLM — SLO validation under load	TTFT/ITL P99 SLO from step 2	Experimental confirmation of the real throughput per GPU
Capacity planning for on-premise inference	Demand profile and N GPUs from step 4	Scaling policy and autoscaling triggers
Cost per token and per request	EUR/GPU-hour from step §2.2	EUR/1M tokens per model and batching
Cloud GPU commitment and spot	Cloud prices from table §2.4	Optimisation of the complementary cloud tier
Sovereign on-premise vs hyperscalers	Break-even from §3	Final buy/rent decision including the sovereignty axis
TCO of an on-premise GPU cluster	Capex and opex from §2.1	Detailed TCO model with every line item
GPU utilisation as FinOps	Target utilisation from step 2	Scheduling levers to raise real utilisation

Sources

Spheron · LLM Inference SLO Engineering: TTFT, ITL, and P99 Latency Budgets for Production AI (2026) — https://www.spheron.network/blog/llm-inference-slo-ttft-itl-latency-budget-guide-2026/
Spheron · vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026) — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
Spheron · GPU Cloud Pricing 2026: H100 from 1.03 USD/hr, B200 from 2.12 USD/hr — https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
MLPerf Inference v5.1 — Red Hat: 5 777 tok/s (offline) en Llama 3.1-8B FP8 en H100 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
MLPerf Inference v6.0 Results Explained: GPU Performance Rankings for AI Workloads (2026) — https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
IntuitionLabs · H100 Rental Prices Compared: 1.49–6.98 USD/hr Across 15+ Cloud Providers (2026) — https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison
CloudZero · Cloud GPU Pricing Comparison: AWS vs Azure vs GCP For AI Workloads (2026) — https://www.cloudzero.com/blog/cloud-gpu-pricing-comparison/
Lenovo Press · On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing 2026: Rent vs Buy Cost Analysis — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Red Hat · 233 % 3-year ROI and 13 months to payback with Red Hat AI (feb. 2026) — https://www.redhat.com/en/blog/233-3-year-return-investment-and-13-months-payback-red-hat-ai
VentureBeat · 5 % GPU utilization: the 401 billion USD AI infrastructure problem — https://venturebeat.com/infrastructure/5-gpu-utilization-the-401-billion-ai-infrastructure-problem-enterprises-cant-keep-ignoring/
DZone · Queueing Theory for LLM Inference — https://dzone.com/articles/queueing-theory-for-llm-inference
GuideLLM · Evaluate LLM deployments for real-world inference (Red Hat Developer, jun. 2025) — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference

Parameter	Symbol	Example
Requests per day (average)	\(D\)	5,000,000
Input tokens per request (average)	\(L_{\text{in}}\)	512
Output tokens per request (average)	\(L_{\text{out}}\)	256
Peak factor (peak hour vs average ratio)	\(k_{\text{peak}}\)	1.8
Hours per day with significant traffic	\(H\)	16

Synthesis on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

TL;DR

You are here: every stage at once

The analogy: forensic analysis of a request

The scenario: a multi-tenant support chatbot for regulated customers

Step 0 — The frame: the answer the user sees

Rewinding backwards: what was already in place before the request

t = −90 days — The previous Retrain stage closes the earlier cycle

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

t = −31 days — The Deploy stage: the v7 adapter enters serving

Moving forward: the user’s request crosses the system

t = 0 ms — Entry through the gateway

t = 8 ms — Pulling the versioned prompt

t = 12 ms — RAG retrieval

t = 40 ms — Building the final payload

t = 45 ms — Prefill

t = 580 ms — Decode (streaming)

t = 4,200 ms — Complete answer, root span closed

In parallel: Observe is watching

The feedback: the loop closes

Insertion into feedback_signals

Triage by root cause

The next cycle picks it up

What goes in each trace: identity and traceability

Synthesis diagram: how the pieces fit

The on-premise stack applied

Ten bridges between stages where the system breaks

How to walk through the blog

What we have not covered (yet)

See also

References

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

TL;DR

Why the three axes must be measured together

Architecture of the integrated bench

The Kubernetes Job: the full YAML

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

AIPerf — NVIDIA’s concurrency profiler

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

LiteLLM — the per-request token counter

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

Kepler — pod-level energy

The MLPerf Power reference

Unifying the metrics: the PromQL queries

The 3-axis scorecard: table and interpretation

The multi-objective Pareto frontier

Reproducible design: the twelve mandatory metadata

Idempotence and versioning of results

How it connects with the other posts in the series

Full flow of a benchmarking session

Cost of the experiment

What the harness does not measure (and why)

Sources

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

TL;DR

1. The sizing chain: from demand to the number of GPUs

1.1 Step 1 — Characterise the demand

1.2 Step 2 — Set the SLO and derive the maximum utilisation

1.3 Step 3 — Required throughput and throughput per GPU

1.4 Step 4 — Number of GPUs and number of nodes

1.5 Summary of the chain

2. From sizing to cost: the TCO model

2.1 Annual cost per node (base scenario)

2.2 From €/node-year to €/GPU-hour

2.3 From €/GPU-hour to €/1M tokens

2.4 Comparison with cloud (equivalent €/1M tokens)

3. Break-even and payback of your own investment

3.1 The utilisation break-even formula

3.2 Capex payback

3.3 Simple payback (capex only vs equivalent cloud)

4. Decision table: when to buy, rent or go hybrid

5. Sensitivity analysis

5.1 Sizing vs hourly profile and peak factor

5.2 Break-even vs sustained utilisation