Llmops-Pipeline on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

Fri, 22 May 2026 16:00:00 +0200

TL;DR

Across several series the blog has laid out the pieces that hold up an LLM system in production: the Data stage (dataset versioning, ingestion and vector stores, RAG over Kafka), the Tune stage (continuous fine-tuning), the Eval stage (evals as the layer after tracing, guardrails and safety), the Deploy stage (KV cache, PagedAttention, disaggregated serving, multi-tenant GPU cluster, vLLM on Kubernetes, LLM operators on K8s), the Observe stage (tracing with AgentSight, MCP observability, eBPF + drift), the Retrain stage (closing the feedback → dataset → adapter loop), and the cross-cutting components (prompt versioning and data versioning). What is missing is joining them up: seeing a single request cross every piece in order, in one coherent story. That is what this post does. We take a specific request from a multi-tenant support chatbot, rewind it back to the data that trained the adapter serving it today, follow it forward through serving, watch it reach the feedback store when the user marks a thumbs-down, and leave it as the seed of the next quarterly retrain cycle. The route works as a mental map and as an integrator’s guide: the system does not hold up if a single one of the seven pieces (six stages + two cross-cutting) is broken or absent. The practical lesson of the tour is not a new one. It is that everything is connected, that local measurements lie when taken in isolation, and that the real cost of not operating one stage well is paid by another stage further down the line.

You are here: every stage at once

Unlike previous posts, where the mini-map marked a single active box, this one covers the whole pipeline. It is the only post on the blog that activates the six stages and the two cross-cutting components simultaneously, because we follow a real request that crosses all of them.

The analogy: forensic analysis of a request

When an air accident happens, the forensic analysis does not limit itself to the last seconds of the flight. The investigation team rewinds to the maintenance of the previous six months, the manufacturer’s protocols, the pilot’s record, the weather briefing, the controller’s decisions, the incident history on the same model. The conclusion is rarely “the wing broke”; it is “the wing broke because an inspection protocol written a certain way did not detect microcracks that the 2014 calculation model did not consider critical and that were critical beyond a certain fatigue cycle”.

When an LLM request in production fails or succeeds, there is also a long causal chain behind it. The answer the user sees is the last frame; what determined it starts months earlier and branches across six operational stages. If you only look at the last frame, you attribute the result to the model. If you look at the whole chain, you see that the model is one of twelve factors and rarely the most important.

This post does that forensic analysis, but the other way round: instead of starting from a failure and rewinding, we start from a specific request that works and break down what had to happen for it to work, and what will happen to it afterwards. It is a guided tour, not an incident diagnosis. But the mental discipline is the same: no stage is autonomous, and understanding the system means understanding the bridges between stages, not just the boxes.

The scenario: a multi-tenant support chatbot for regulated customers

For the tour we use a concrete realistic scenario, representative enough that the observations carry over to most serious deployments in May 2026. It is an LLM customer support SaaS product, sold to several corporate customers (multi-tenant) in regulated sectors (banking, insurance, health). The product:

Accepts natural-language questions through a chat embedded in the customer’s website.
Retrieves relevant fragments from the customer’s internal knowledge base (product documents, contractual terms, FAQs).
Answers citing the fragment used, in the user’s language, with a tone adapted to the brand.
If the question requires an action (cancel, change plan, open a case), it invokes a tool from the catalogue exposed via MCP, with user confirmation.
It is contractually obliged not to invent customer data, to respect the tenant’s style guide, and to meet ENS / NIS2 compliance on data in the EU.

The infrastructure is on-premise over Kubernetes (RKE2 with the Cilium CNI, in the pattern covered in Cilium BGP and CNI networking). The GPU plane is a generic 4×H100 SXM cluster (80 GB each, NVLink) for serving and production training, plus an RTX 4090 (24 GB) for development, offline evals and ad-hoc re-embeddings.

The request we are going to follow:

User (end customer, Spanish, premium plan with an insurer): “How do I cancel my premium subscription? I have been paying for three months and I have not used half of the cover.”

At first sight, a simple question. Let us see what is behind it.

Step 0 — The frame: the answer the user sees

At 11:47:32 on a Tuesday, the user sends the message. 4.2 seconds later, this appears in the UI:

To cancel your premium subscription, follow these steps: (1) Sign in to the customer portal with your ID number and password. (2) Go to My policies → Premium Life 2024. (3) Press Request cancellation. (4) Select the reason (in your case, “not using the product”) and confirm. You will receive a confirmation email within 24-48 h.

Before you cancel, let me remind you that your plan includes a free annual medical check-up and 24/7 travel assistance, cover that is lost on cancellation. If you like, I can help you review your use of the cover over the past three months to see whether switching to a more tailored plan suits you better than cancelling.

Source: [Cancellation policy, section 4.2 of the 2024 General Terms]

After reading it, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”.

That sequence, question, answer, thumbs-down with structured feedback, is the last visible frame. Let us go backwards to understand what had to happen for the answer to come out that way.

Rewinding backwards: what was already in place before the request

Before the user typed, the system already had a model loaded in serving, an active prompt labelled production, an up-to-date vector index, a versioned dataset from the last fine-tuning, and a golden eval set that validated the promotion. Each of those artefacts got there through a process. We go through four jumps backwards.

t = −90 days — The previous Retrain stage closes the earlier cycle

Three months ago, during a quarterly Retrain cycle, two things happened. The first: the support team reviewed the feedback accumulated over the previous six months and saw a pattern, the model answering premium users in an excessively formal tone, with users reporting that it “feels robotic”. The second: a one-off incident (a customer cancelling over an answer perceived as curt) triggered an incident-driven mini-cycle.

The process, covered in detail in the Retrain post, followed five sub-processes:

Feedback capture — explicit thumbs-down plus implicit feedback (abandonments, retries) accumulated in a feedback_signals table in Postgres, all of them with a trace_id that allows rewinding to the exact context.
Triage by root cause — the “curt tone” incident cluster was categorised as a prompt issue (it was not the model answering badly, it was the system prompt asking for an overly formal register). One sub-cluster was a model issue (in some cases the model dug its heels in even with a warmer prompt).
Dataset enrichment — the team manually annotated 280 cases where the model was too curt, labelled with the reference answer (“how it should have answered”). Double annotation on the critical 20%; cases with a quality score below 4 were left out.
Cadence decision — the incident was treated as incident-driven; the rest of the quarterly Retrain followed the calendar.
Promotion — the new customer_support_v7 adapter went through eval gates against customer_support_v6, a 5% canary for a week, and was promoted when the golden set metrics showed a stable improvement in the “tone / clarity” segment with no regressions elsewhere.

The result: the adapter active in production when the user sent the Step 0 request is customer_support_v7, trained on the enriched dataset enriched_retrain_2026_q1 version 3, with double lineage back to the original incident.

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

Immediately after Retrain, the Data stage of the LLMOps pipeline does its work. Three critical operations, covered in detail in the data versioning post:

Immutable versioning of the enriched dataset with DVC, sha256 hash propagated to the registry. The identifier (enriched_retrain_2026_q1, v3, sha256:9af...) becomes the luggage tag that will travel through the coming stages.
Schema contract validated by CI: every row meets the JSON Schema of the entry the trainer expects (example_id, input.user_query, input.retrieved_context, expected_output, rubric, segment, difficulty). A validation fails in CI if any row breaks the contract.
Holdout segregation check: the normalised sha256 hash of each input is compared against every hash in the active golden eval set (customer_support_golden_v12). Zero overlaps = the dataset does not contaminate the eval. Had there been a single one, CI would have blocked the merge.

In parallel, the RAG corpus (product manuals, FAQs, general terms of the insurer tenant) is kept alive. The ingestion pipeline keeps capturing changes from the customer’s CMS: a new section of the cancellation policy was modified in February and reindexed in Qdrant. As the post about RAG over Kafka explains, the corpus is not retrained with every change: only the delta is re-embedded, and lakeFS keeps a branch of the embeddings bucket with the new version. The branch is merged into main when recall@10 over a representative set of queries stays above the threshold (0.78 in this system).

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

Three weeks after closing the dataset, training of the new LoRA adapter starts. As the continuous fine-tuning post details, the production pattern in 2026 avoids retraining the base model, which is expensive, slow and irreversible, and favours a LoRA adapter over a stable base model (in this system, Llama 3 70B-instruct quantised to INT8 for serving). The training:

Runs over 4 of the H100s (NVLink, tensor parallel) for ~6 hours.
Uses transformers + PEFT + bitsandbytes, monitored by MLflow.
Every step records the dataset_id, dataset_version and dataset_hash as an input artifact in MLflow.
The output, a customer_support_v7.safetensors file of ~280 MB with the LoRA weights, is uploaded to MinIO with its own hash, and MLflow records model_id, model_version, parent_dataset.

At this point the lineage chain is closed for this leg:

enriched_retrain_2026_q1, v3, sha256:9af...
│
▼
mlflow run train, run_id: 0xa721...
│
▼
customer_support_v7, sha256:5c1...

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

The freshly trained adapter is not promoted. It goes through an eval suite covered in detail in the post about evals. The golden eval set, customer_support_golden_v12, 850 human-curated examples with an inter-annotator kappa of 0.81, is run against two models: the candidate v7 adapter and the v6 currently in production. The metrics:

Metric	v6 (prod)	v7 (cand.)	Threshold
Faithfulness to the RAG fragment	0.87	0.89	≥ 0.82
Toxicity (low is good)	0.012	0.011	≤ 0.02
“Warm but professional” tone (judge LLM)	0.71	0.84	≥ 0.78
Format compliance (structured markdown)	0.94	0.93	≥ 0.90
Helpful-but-not-pushy (judge LLM)	0.66	0.79	≥ 0.75
Latency p95 (ms)	2,840	2,910	≤ 3,500

To this is added the guardrails and safety suite covered in the guardrails post: jailbreak resistance, PII leakage detection, prompt injection over MCP tools. The v7 improves on safety in two metrics and ties on the rest.

The v7 enters a 5% traffic canary for 7 days, under close monitoring. At the end of the canary, the online metrics confirm what the offline ones anticipated: an improvement in tone and helpfulness, equivalent latency, no new failure modes. Promotion approved. The v7 moves to the production label.

t = −31 days — The Deploy stage: the v7 adapter enters serving

The customer_support_v7 adapter is promoted to the serving cluster. Three pieces covered in separate posts come into play.

vLLM as the inference engine. The engine lives on Kubernetes, deployed via a dedicated Operator, as the post about LLM operators and the post about vLLM on K8s explain. The operator is responsible for detecting the new adapter in the registry, hot-loading it without restarting the engine (a native vLLM capability with --enable-lora), and directing traffic based on the label.

Disaggregated serving. As the post about disaggregated serving details, the system separates prefill (compute-intensive, throughput-bound) and decode (memory-intensive, latency-bound) into different GPU pools. The user’s request, when it arrives, prefills in a specialised pod and decodes in another, communicating over NVLink plus a shared KV cache fabric.

Multi-tenant GPU cluster. The H100 cluster serves several tenants, not just the insurer from Step 0. As the post about the multi-tenant cluster explains, isolation is realised on four planes: the Kubernetes namespace, ACLs over adapters (only the tenant’s namespace loads its LoRAs), partitioning of the KV cache per tenant (one tenant cannot read another’s cached prefixes), and a tokens-per-minute quota enforced at the gateway.

Synchronised prompt registry. The product’s system_prompt lives in Langfuse with the production label. The active version is customer_support_system_prompt, version 12. The gateway reads the prompt from Langfuse on the request path (with a few seconds of cache so it does not hammer the registry). Detailed in the prompt versioning post.

The result at t = −31 days: the combination (adapter v7, prompt v12, golden v12) is active and served. The system is ready for the request that will arrive 31 days later.

Moving forward: the user’s request crosses the system

Back to Step 0: 11:47:32 on a Tuesday. The user presses Enter. We go in real time, in milliseconds.

t = 0 ms — Entry through the gateway

The user’s browser POSTs to chat.aseguradora-ejemplo.com/api/chat. The traffic crosses the edge load balancer and enters the SaaS product’s API gateway. The gateway:

Authenticates the user’s JWT (an end customer of the insurer tenant).
Extracts the tenant_id and checks that its tokens-per-minute quota is not exhausted.
Resolves which model_id, adapter_id and prompt_id correspond to this tenant and product. In this case: llama-3-70b-int8 + customer_support_v7 + prompt label production.
Builds a unique trace_id (W3C TraceContext, propagable to OTel) and starts a root span.

At 8 ms, the gateway passes the request to the prefill pool.

t = 8 ms — Pulling the versioned prompt

Before serving, the OpenAI-compatible client the engine uses internally pulls the active system prompt. As the post about prompt versioning details, the pattern is:

prompt = prompt_registry.pull(
 name="customer_support_system_prompt",
 label="production", # now pointing at v12
)
# A local 30 s cache reduces the round-trip to 0.1 % of requests

The OTel span of the prompt pull carries the attributes gen_ai.prompt.id = customer_support_system_prompt, gen_ai.prompt.version = 12, gen_ai.prompt.label = production. They are propagated to all children.

t = 12 ms — RAG retrieval

The system needs context from the tenant’s knowledge base. It runs:

query_embedding = encoder.encode(user_query)
chunks = qdrant.search(
 collection=f"tenant_{tenant_id}_kb_v3",
 vector=query_embedding,
 limit=4,
 score_threshold=0.72,
)
reranked = reranker.rerank(user_query, chunks, top_k=2)

At 38 ms, the reranker returns two fragments: one from the Cancellation policy, section 4.2 and another from Premium plan benefits, section 2.1. As the post about PostgreSQL + Qdrant details, the tenant’s corpus is kept isolated by collection and ACL: no tenant can read another’s chunks.

t = 40 ms — Building the final payload

The engine composes:

[system_prompt v12]
+ [retrieved context: 2 chunks]
+ [short session history: 1 previous turn]
+ [user query]

Total: ~1,850 tokens of context. The OTel span records gen_ai.request.input_tokens = 1850, gen_ai.request.model = llama-3-70b-int8, gen_ai.request.adapter = customer_support_v7.

t = 45 ms — Prefill

The payload enters the prefill pool. The GPU processes the 1,850 tokens in a single parallel pass, computing for each token its K and V vectors (attention key and value). Those vectors are materialised as the KV cache, covered in detail in the KV cache fundamentals post. The resulting cache takes up ~120 MB of VRAM in INT8.

Here a key optimisation appears: the v12 system prompt is cached in the prefill pool (prefix caching, covered in the post about PagedAttention). Since the system prompt is the same for this tenant, the first ~500 tokens of the context are not recomputed: they are read from the prefix cache. That reduces the effective prefill from 1,850 tokens to ~1,350 tokens, saving ~270 ms of compute.

At 580 ms (effective prefill), the TTFT (time to first token) is ready. The first token leaves for the decode pool.

t = 580 ms — Decode (streaming)

The decode pool receives the prefilled KV cache and starts generation token by token. As the post about disaggregated serving details, the prefill/decode separation is what lets a multi-tenant system keep TPS stable: the decode pool is sized to sustain thousands of sessions decoding in parallel at a low cost per token, while the prefill pool is sized for bursts of short TTFT.

Generation at ~80 tokens per second. The answer will be ~290 tokens. Total decode time: ~3.6 s. Streaming: the user starts seeing words from t = 580 ms.

While decode advances, the engine emits child spans on every iteration with gen_ai.response.tokens_generated, gen_ai.response.cache_hit_ratio, gen_ai.response.cumulative_latency. The post about AgentSight and the post about MCP observability with OTel cover the detailed instrumentation of this layer.

t = 4,200 ms — Complete answer, root span closed

Generation finishes. The engine closes the root span with gen_ai.response.completion_tokens = 290, gen_ai.response.finish_reason = stop, gen_ai.response.total_latency_ms = 4200. The user sees the final answer. The session is ready for a next turn or for the user to click thumbs-up/thumbs-down.

By this point, every active stage has taken part:

Data (pre-existing): the indexed RAG corpus, the dataset that trained the adapter, the golden set that validated it.
Tune (pre-existing): the v7 adapter trained 45 days ago.
Eval (pre-existing): the gates that approved the promotion.
Deploy (at this very moment): vLLM + disaggregated + KV cache + multi-tenant.
Observe (at this very moment): the OTel spans emitted to Langfuse + Tempo, the metrics to Prometheus.
Retrain (about to activate): the feedback the user will leave in 15 seconds.

In parallel: Observe is watching

While the request happens, several pieces of Observe run in parallel and leave a structured trail.

OTel tracing. Every span (gateway, prompt pull, retrieval, prefill, decode) travels to Langfuse and to an OTel collector that forwards them to a backend (Tempo / Jaeger). The unique trace_id links all the spans. As the post about tracing with AgentSight details, end-to-end propagation is the main enabler of post-incident debugging: without it, you cannot reconstruct what happened three weeks later.

Runtime metrics. The engine emits Prometheus metrics per interval: gpu_utilization, kv_cache_usage, tokens_per_second, queue_depth, prefill_latency_p95, decode_latency_p95. The metrics are not tied to a trace; they are aggregated by tenant and service.

Online LLM-as-judge. A configurable percentage of answers (2% in this system) is also run through a judge LLM in the background, which scores the answer against a simple rubric (correct / partial / incorrect + a tone score). The judge does not block the answer to the user; it feeds the dashboard.

Statistical drift. In parallel, a slower pipeline computes drift over the distribution of inputs and outputs. As the post about eBPF + drift explains, low-level monitoring (latency, error rate per endpoint) is complemented with statistical drift detection (KS test, embedding distance) that spots when “something is wrong” before a thumbs-down confirms it.

Safety and guardrails monitor. The guardrails post describes the layer that watches for jailbreak attempts, PII leakage and prompt injection via MCP tools. In this case, none of them fires.

All these pieces operate continuously, not per request. But this particular request left its trail in every one of them.

The feedback: the loop closes

Fifteen seconds after reading the answer, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”. That apparently trivial gesture triggers an important sequence.

Insertion into feedback_signals

As the post about Retrain details, the thumbs-down is persisted as a structured row in a Postgres table:

INSERT INTO feedback_signals (
 signal_id, trace_id, request_id, signal_type, signal_value,
 prompt_id, prompt_version, model, user_segment, occurred_at
) VALUES (
 gen_random_uuid(),
 '4f5...', -- the trace_id from Step 0
 'r-22a...', -- request_id
 'thumbs',
 '{"vote":"down","reason":"too pushy","text":"I only wanted to know how to cancel"}',
 'customer_support_system_prompt',
 12,
 'llama-3-70b-int8+customer_support_v7',
 'premium-es',
 '2026-05-19T11:47:51+02:00'
);

With this, the row is linked by trace_id to everything that happened: prompt v12, retrieved context, complete output, latency metrics, judge score (0.82 in this case, considered good by the judge, though the human disagrees).

Triage by root cause

The MLE team runs triage the next morning. Combining heuristic rules, LLM-as-classifier and human review:

The signal is not a model issue: the model answered correctly to the prompt it received.
It is not a retrieval issue: the retrieved chunks were the right ones.
It is not an infra issue: latency was normal.
It is a prompt issue: the v12 system prompt instructs the model to “offer alternatives before processing destructive actions”. That instruction generates the “salesy tone” in some contexts.

The incident accumulates with others from the month in the “salesy tone” cluster. When the cluster crosses a threshold (typically 30-50 incidents of the same type or a percentage of the total), it will enter an incident-driven mini-cycle or wait for the quarterly Retrain, depending on the size.

The next cycle picks it up

Three months later, in the next quarterly Retrain, this feedback is one of many that will motivate two changes:

A new prompt version v13 with an adjusted instruction: “offer alternatives only if the user does not express a clear intention to cancel”.
A possible reinforcement of the adapter with cases of a more direct tone for premium-es. If the cluster justifies it.

The v13 will go into its own eval gate. The golden set will grow with cases where the correct tone is “direct, not salesy”. The v8 of the adapter (if it arrives) will retrain over the enriched dataset enriched_retrain_2026_q2 that already contains this annotated case.

The loop closes. The Step 0 request has contributed to the version of the system that will serve another user three months later.

What goes in each trace: identity and traceability

If the reader looks at the seven identifiers that are everywhere in this route, they see the network of identities that makes all of the above possible. It is the identity infrastructure of an LLM system in production:

trace_id 4f5... (unique per request)
request_id r-22a... (idem)
prompt_id customer_support_system_prompt
prompt_version 12
prompt_label production
dataset_id enriched_retrain_2026_q1
dataset_version v3 (sha256:9af...)
model_id llama-3-70b-int8
adapter_id customer_support_v7 (sha256:5c1...)
deployment_id d-prod-7b
schema_version 3.2
tenant_id aseguradora-ejemplo
user_segment premium-es
golden_set_id customer_support_golden_v12

If a single piece of that set is missing or does not propagate, the chain breaks. The next incident investigated will land in “we cannot rewind to the origin because the system did not record it”. That is why the cross-cutting components, prompt versioning and data versioning, are not luxuries: they are the connection without which the other six stages operate blind.

Synthesis diagram: how the pieces fit

 ┌─────────────────────────────────────────┐
│ User (end customer, B2C) │
└─────────────────┬───────────────────────┘
│ chat msg + JWT
▼
┌─────────────────────────────────────────┐
│ Edge LB + WAF + Cilium CNI │
└─────────────────┬───────────────────────┘
│ HTTPS, internal mTLS
▼
┌─────────────────────────────────────────────────┐
│ API Gateway (auth, quota, model routing) │
│ - Resolves tenant → model + adapter + prompt │
│ - Starts trace_id (W3C) │
└──────┬─────────────────────┬────────────────────┘
│ │
(pull prompt) │ │ (pull config)
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ Langfuse Prompt │ │ Model registry │
│ Registry (v12) │ │ (adapter v7) │
└─────────┬──────────┘ └──────────┬───────────┘
│ │
└──────────┬───────────────┘
│ payload ready
▼
┌──────────────────────────────────────────┐
│ vLLM engine (K8s Operator) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pool prefill │ → │ Pool decode │ │
│ │ (H100×N) │ │ (H100×M) │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ KV cache fabric │ │
│ └──────────────────┘ │
│ - prefix caching of the system prompt │
│ - PagedAttention │
└──────┬───────────────────────────────────┘
│ tokens stream
▼
┌─────────────────────────────────────────┐
│ User sees the answer + thumbs/UX UI │
└─────────────────┬───────────────────────┘
│ feedback (15 s later)
▼
┌─────────────────────────────────────────┐
│ feedback_signals (Postgres) │
│ + Langfuse scores │
└─────────────────┬───────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
triage quarterly Retrain cycle dataset_id
root cause or incident-driven enriched (DVC)
│
▼
Tune of v8
(next cycle)
In parallel throughout the request, OTel instrumentation:
spans → Tempo / Jaeger ; events → Langfuse ; metrics → Prometheus

The on-premise stack applied

Taking the above to a generic on-premise consultant-profile infrastructure (RTX 4090 + 4×H100 SXM cluster):

Layer	Typical resources
Network plane	Edge LB (HAProxy / nginx ingress) + Cilium CNI with BGP, covered in Cilium BGP
K8s compute plane	RKE2 with two manager nodes + a GPU node pool
Production GPU plane	4× H100 SXM (NVLink, 80 GB each), partitioned via MIG into prefill/decode pools
Development GPU plane	1× RTX 4090 (24 GB) for offline evals, drift-check embeddings, smoke tests
Storage plane	MinIO or Ceph object store; DVC remote + lakeFS backend
OLTP data plane	Postgres 18 with replication; pgvector 0.8 for small cases
Vector plane	Qdrant or Milvus for large RAG corpora
Stream plane	Kafka (Redpanda / pure Apache) + Schema Registry; CDC with Debezium or Flink CDC
Observability plane	OTel Collector + Tempo (traces) + Prometheus (metrics) + Loki (logs); Langfuse for the LLM-specific part
Runtime security plane	Tetragon, covered in the runtime security post

The real density is not the sum of the boxes: it is the operations that tie the boxes together. A cluster with every piece but no versioning discipline, no end-to-end trace_id propagation, no schema contracts and no cadenced retraining is a cluster that serves LLM once and then ages. The difference between a project and a platform is exactly that.

Ten bridges between stages where the system breaks

The route reveals something important: failures are rarely inside a stage; they are on the bridges between stages. Ten common bridges:

Data → Tune: the dataset does not propagate its (dataset_id, dataset_version) to the trainer. The same dataset trained twice produces two model_id values that cannot be told apart.
Tune → Eval: the trained model does not propagate its lineage to the eval run. The eval passes, but there is no record of which dataset it was trained on. Three months later, irreproducible.
Eval → Deploy: promotion happens without the serving system recording which version of the adapter it is serving at each moment. The day the model gives a dangerous answer, nobody knows which adapter answered.
Deploy → Observe: the engine does not emit gen_ai.request.adapter, gen_ai.prompt.version, gen_ai.dataset.version as span attributes. The traces exist but cannot be crossed with the lineage.
Observe → Retrain: the feedback is captured in a tool (Langfuse, Phoenix) but nobody reads it. The Retrain stage “exists”, but feedback piles up untriaged.
Retrain → Data: the enriched dataset goes into the next Tune without passing through versioning discipline, schema contract and holdout check. Silent contamination of the golden set.
Prompt versioning ↔ everything: the prompt_id, prompt_version does not propagate to the spans. The day the team discovers that a prompt change regressed the system, it cannot isolate which one or when.
Data versioning ↔ everything: the dataset_id, dataset_version does not show up in experiment tracking. “v8 is retrained” but nobody can prove it was over the enriched dataset and not the old one.
MCP ↔ tools: the system invokes tools (cancellation, policy changes) but does not record a gen_ai.tool.invocation_id linked to the trace. The actions end up dissociated from the answer that generated them.
Schema Registry ↔ data: datasets version content but not schema. A breaking change in expected_output breaks the eval silently; nobody notices anything until a human reviews the results.

The bridges are covered throughout the blog. Operations enforce them. The team’s culture keeps them.

How to walk through the blog

If you arrive at this post from outside and want a reading route:

The map: The six-stage LLMOps pipeline — the master map of everything else.
The context: MLOps specific to LLMs in 2026 — the landscape and why LLMOps is not classic MLOps.
Inference from the inside out: KV cache → PagedAttention deep dive → Disaggregated serving → Multi-tenant GPU cluster → vLLM on K8s → LLM operators on K8s.
Data: Data versioning with DVC and lakeFS → PostgreSQL + Qdrant ingestion → RAG over Kafka.
Tune: Continuous fine-tuning in production.
Eval: Evals: the layer after tracing → Guardrails and safety.
Observe: AgentSight LLM tracing → MCP observability with OTel → On-device eBPF + drift.
Retrain: Closing the feedback → dataset → adapter loop.
Cross-cutting: Prompt versioning with Langfuse and MLflow.
Supporting infrastructure (the base everything is built on): RKE2 with Cilium BGP, Hubble + eBPF observability, Tetragon runtime security.

What we have not covered (yet)

At the first level the main things are there. The following posts on the blog, when the topics justify it, could go deeper into:

Schema Registry for LLM data and prompts: the other half of the data contract.
A dedicated AI Gateway: LiteLLM, Portkey, Kong AI Gateway as a control plane.
OTel gen_ai semantic conventions: the emerging standard that ties the seven identifiers of the “identity” block into well-formed spans.
Federated learning over regulated customer data: how to train without centralising the corpus.
Capacity planning for shared multi-tenant clusters.
Disaster recovery of an LLM service: how to reproduce the state of the system 30 days back.
Cost accounting per tenant: tokens × weights × adapter × infrastructure → invoice.

References

W3C Trace Context — end-to-end propagation of traceparent and tracestate.
OpenTelemetry GenAI Semantic Conventions — gen_ai.* attributes for LLM spans.
Langfuse documentation — observability and prompt registry.
vLLM documentation — production inference engine with PagedAttention and LoRA hot-swap.
Kubernetes Operators — the declarative management pattern.
MLflow Tracking and Model Registry — lineage of runs and input artifacts.
DVC and lakeFS — dataset versioning, unified in Nov 2025.
OpenLineage — an open standard for lineage events between systems.
ENS / NIS2: compliance frameworks that apply to operators in the EU; recommended reading for the context in which the scenario operates.

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

Sat, 23 May 2026 07:30:00 +0200

TL;DR

For each of the six LLMOps stages (Data, Tune, Eval, Deploy, Observe, Retrain) and the two cross-cutting components (prompt + data versioning), the open source ecosystem has canonical pieces that this blog has been citing again and again. This post gathers them in one place with cards of roughly 150 words per core tool: what it does, how it differs from the alternatives inside the same bucket, its licence and governance model, and a typical gotcha that you only learn in production. Plus more alternatives as bullets, a decision matrix per stage according to the case (small / large corpus, single tenant / multi-tenant…), a diagram of the connected OSS stack and a master table of licences / EE offerings. The intention: that the reader closes the post knowing what is available, which company maintains it, what gap each piece fills, and when to choose it. This is not opinion: it is a curated catalogue.

You are here: every stage, but by OSS column

This post shares a map with the two previous ones in the series, the six stages and the two cross-cutting pieces are all active, but it zooms in on the open source column.

The analogy: the electrician’s toolbox

A professional electrician arrives at an installation with a box organised into compartments. Nothing is improvised: for every type of cable there is a specific wire stripper, for every screw a screwdriver of the exact gauge, for every measurement a multimeter and a clamp meter, for every connection the right terminal block or connector. The difference between a professional electrician and a handyman is not that one knows more theory, the handyman has often read the manuals, it is that the right tool is within arm’s reach and he knows when to use each one. The day the specific stripper is missing, improvising with a box cutter breaks the insulation, leaves a badly terminated cable, and the panel comes back under warranty two months later.

The OSS LLMOps stack works the same way. For every canonical problem, versioning a dataset, indexing a corpus for retrieval, serving tokens with dynamic batching, propagating trace_id end-to-end, managing prompts with a production label, orchestrating retraining pipelines, there is a canonical piece of the open source ecosystem that solves it, maintained by a serious community or foundation, with a clear licence and a well documented gotcha. The consultant who knows which tool to use for each job builds a robust system in weeks; the one who improvises with “whatever the team already knows” pays later in operations, usually once the system is carrying real load and any replacement is expensive.

This post opens the toolbox and shows each card. It is not a user manual, that is what the deep-dive posts linked at the end are for; it is the curated catalogue.

Diagram of the connected reference OSS stack

The catalogue makes sense once you see how the pieces connect into a single coherent architecture, the one this blog has been describing throughout the series. The boxes do not float; they talk to each other over stable contracts (HTTP, gRPC, OTel, Kafka, S3/MinIO API).

Solid arrows mark data / control flow; the dashed blue ones are OTel traces. The K8s plane holds everything up. The control plane at the bottom is where the retraining pipelines, the evals in CI, the versioned prompts and the lineage live. The data plane on the left feeds both the serving side (RAG, configs) and the control plane (datasets, lineage). The observability plane receives from serving and from everything else.

Now we go stage by stage. Each one opens with a context paragraph, then cards for the core tools (about 150 words each), bullets with relevant alternatives, and a specific decision matrix at the end.

Stage 1 — Data + cross-cutting Data versioning

The Data stage solves three distinct problems that beginners confuse: versioning datasets (so that (dataset_id, version, hash) exists and propagates), storing and serving the operational corpus (object store + vector index + structured text), and moving it between systems with CDC and stable schemas. Covered in detail in the posts on data versioning with DVC and lakeFS, PostgreSQL + Qdrant in ingestion and RAG over Kafka.

DVC (Data Version Control)

DVC puts datasets under version control with the same discipline that git applies to code. The .dvc pointers live in git (plain text, about 200 bytes per dataset), the bulk content lives in a remote object store (S3, MinIO, Azure Blob, GCS). Each dvc add computes a SHA-256 hash of the dataset, uploads it to the remote and stores the pointer. The key line: the dataset_hash becomes the luggage tag that travels to the trainer, to experiment tracking and to lineage. The same dataset retrained twice produces the same hash, and therefore reproducible experiments. DVC integrates with MLflow and W&B as an input artefact. Gotcha: it works well for datasets that change by replacement (I swap train.jsonl for a new version) and worse for datasets with thousands of small files that change individually. For that case, combine it with lakeFS. Licence Apache 2.0, maintained by Iterative.ai since 2017. There is DVC Studio (managed) and dvc data (pure CLI) on different planes.

lakeFS

lakeFS brings git semantics (branch, commit, merge, rollback) to an entire S3/MinIO/ADLS bucket. Where DVC versions individual files as pointers in git, lakeFS versions the whole bucket: you can branch the corpus, ingest new data into the branch, validate that it passes checks (recall@10 over golden queries for embeddings, completeness for a tabular corpus), and only then merge into main. It is the piece that makes continuous RAG safe: the production corpus is always on main, updates are tested on branches. It offers hooks (pre-merge, pre-commit) that fire automatic validations, and time-travel to reproduce the state of the bucket at a past date. Gotcha: the manifest overhead on huge buckets (hundreds of millions of objects) deserves sizing; lakeFS keeps metadata in its own Postgres, not in the bucket. Licence Apache 2.0, maintained by Treeverse since 2020. Managed offering: lakeFS Cloud.

MinIO

MinIO is the S3-compatible object store that fills the “S3 on-premise” gap without surprises. API identical to S3 (the AWS SDKs work by pointing them at a different endpoint), its own CLI client (mc), erasure-coded mode for fault tolerance, bucket-to-bucket replication, encryption at rest. It is the base on which the other data plane components are built: DVC remote, lakeFS underlying storage, Postgres snapshots, MLflow artefacts, eval datasets, saved models, distributed KV cache fabric. In small deployments it runs single-node multi-disk; in serious ones, distributed clusters. Gotcha: the licence changed to AGPLv3 in 2021 (it was Apache 2.0 before), which means distributing software connected to MinIO obliges you to open the code that connects to it. For internal on-premise use this is not a problem; for a vendor packaging MinIO inside a commercial product, it is. Maintained by MinIO Inc. with a SUBNET enterprise offering and a community fork called AIStor launched in 2025.

Qdrant

Qdrant is the OSS vector database most aligned with this blog’s “RAG corpus per tenant with strict ACLs” pattern. Written in Rust, it exposes a REST + gRPC API, indexes with HNSW + scalar/binary quantisation to cut memory, supports efficient payload filtering (it is not post-filtering: it integrates the filter into the HNSW search), and allows collections isolated per tenant. For the multi-tenant chatbot scenario, Qdrant is where the tenant_<id>_kb_v3 collections with strict ACL live. It scales well horizontally (sharding by payload) and vertically (millions of chunks on a node with 64GB RAM). Gotcha: binary quantisation is aggressive, it cuts VRAM by 32× but degrades recall by 10-20 %; enabling it without re-tuning the threshold breaks retrieval silently. Licence Apache 2.0, maintained by Qdrant Solutions GmbH (Germany). There is Qdrant Cloud (managed) and EU-only support for ENS cases.

PostgreSQL + pgvector

Postgres 18 with the pgvector extension is the stack’s “hidden vector database”: when the corpus is small (under a million embeddings) and Postgres is already in production for operational data, running a separate Qdrant is expensive operations. pgvector adds a vector(dim) type, HNSW and IVF indexes, and the operators <->, <#>, <=> for cosine, L2 and dot product. Combined with tsvector (Postgres full-text search) it allows hybrid search, dense + sparse, in a single SQL query. Version 0.8 (2025) introduced halfvec and bit support to reduce size by 4×-8×. Gotcha: HNSW in pgvector consumes a fair amount of RAM to build the index (roughly 2× the size of the embeddings) and blocks inserts during the build; in production you build on a secondary, promote it, and discard the primary. Licence PostgreSQL License (permissive BSD-style) for both the core and pgvector. Maintained by the PostgreSQL Development Group plus pgvector by Andrew Kane + Crunchy Data + Neon.

Apache Kafka + Debezium

Kafka is the event bus where “everything that happens in the company is a stream” becomes real. For LLMOps in production it serves two functions: CDC from source systems (Debezium captures changes in Postgres / MySQL / MongoDB and publishes them as topics) and LLM event buffer (every request, every piece of feedback, every eval result ends up in a topic with the trace_id propagated). As the post on RAG over Kafka explains, the RAG corpus is kept fresh by capturing changes from the CMS / source system as CDC, running the embedding in Flink streaming, and ingesting into Qdrant continuously. Gotcha: a badly sized Kafka with long retention plus multi-client topics turns into a fast disk sink; measuring throughput per topic and key cardinality before production is mandatory. Kafka licence Apache 2.0 (ASF project); Debezium Apache 2.0 (project incubated by Red Hat). Kafka-compatible drop-in alternative: Redpanda (BSL, restricted commercial use).

Apache Flink (brief mention)

Flink processes streams with sub-second latency and exactly-once semantics. On the LLM plane it is used to: run embeddings in streaming (over CDC topics), aggregate online metrics, materialise features for retraining. Licence Apache 2.0, ASF. Common alternative: Spark Structured Streaming (also ASF, micro-batch latency).

More options for Data, mentioned on the blog:

Ceph — object store for large clusters with geo-distributed replication. LGPL/Apache licence, Red Hat / IBM.
Milvus — C++ vector database, an alternative to Qdrant; better for corpora of billions. Apache 2.0, Zilliz.
Karapace — Confluent-compatible OSS Schema Registry. Apache 2.0, Aiven.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage. Apache 2.0, Acryl Data / ASF / Collate respectively.
OpenLineage — cross-system lineage event standard. Apache 2.0, Linux Foundation AI&Data.

Decision matrix — Data:

If your case is	Choose
Corpus < 1M embeddings, you already have Postgres	pgvector (one component fewer)
Corpus 1M-100M, multi-tenant with ACL	Qdrant (integrated filtering, ACLs per collection)
Corpus > 100M, aggressive sharding	Milvus (scales linearly better into the billions)
Training datasets + experiment tracking	DVC over MinIO + MLflow integration
RAG corpus with controlled releases	lakeFS over MinIO + pre-merge hooks
You want both	DVC + lakeFS as complements (the blog’s recommendation)

Stage 2 — Tune

The Tune stage produces a new model_id, model_version, typically a LoRA adapter over a stable base, with lineage back to the dataset and experiment tracking to reproduce it. Detail in the post on continuous fine-tuning.

HuggingFace Transformers + PEFT

transformers is the canonical library for loading and training models of the decoder-only family (Llama, Mistral, Qwen, Gemma…) and encoder-decoder. peft (Parameter-Efficient Fine-Tuning) is the complement that adds declarative support for LoRA, QLoRA, IA3 and assorted adapters. Together they form the mandatory core of the OSS Tune stack: any higher-level framework (Axolotl, LLaMA-Factory) uses them underneath. PEFT allows you to train an adapter of about 280 MB (order of magnitude) instead of a full model of about 140 GB, with a functionally equivalent result on most style / domain adjustment tasks. Gotcha: PEFT with a badly configured target_modules trains an adapter that covers only Q and V of the attention, leaving out key, output proj and MLP. The result looks trained but performs poorly; adding target_modules=["all-linear"] fixes it (at the cost of a larger adapter). Licence Apache 2.0, maintained by Hugging Face SAS (a French company); open governance model with active external maintainers.

bitsandbytes

bitsandbytes implements weight quantisation to 8-bit and 4-bit with NF4 for models loaded with transformers. It reduces the 140 GB of Llama 3 70B FP16 to about 40 GB in NF4, allowing QLoRA training on a single H100 80GB. The trick is that the weights stay quantised in memory while the sensitive computations (attention, gradient updates in the adapter) are done in FP16/BF16 with on-the-fly dequantisation. Ideal for fine-tuning on limited hardware and for serving with vLLM when you want to cut VRAM. Gotcha: NF4 quantisation is lossy; on small models (< 7B) the quality degradation is noticeable. For production serving of models under 7B, INT8 is preferred (more memory, less loss) or FP8 if the hardware supports it (the H100 does). Licence MIT, maintained by Tim Dettmers (originally at U. Washington, now with support from Anthropic and HuggingFace).

MLflow Tracking

MLflow is the reference OSS experiment tracking tool: each trainer run records parameters (lr, batch size, epochs, target_modules), metrics (loss curves, eval scores), artefacts (model, tokeniser, configs) and, crucially, input artefacts (dataset_id, dataset_hash, parent_run). The model registry associates each model_version with a reproducible run_id. The line of continuity between Tune and Deploy passes through here: the deployment reads the model to serve from the registry, with its lineage made explicit. MLflow 2.x integrates MLflow Prompts (prompt registry) and MLflow Tracing (OTel-compatible spans), cutting the number of components needed. Gotcha: the default backend store is SQLite, which works for personal experiments and breaks on a shared cluster. In production: Postgres as the backend store + MinIO/S3 as the artefact store. Licence Apache 2.0, maintained by LF AI & Data (donated by Databricks in 2020).

Axolotl

Axolotl wraps transformers + PEFT + bitsandbytes + DeepSpeed + FSDP in a declarative YAML configuration: instead of writing a script of about 300 lines to configure a fine-tuning run, you define a config.yml with base model, dataset path, LoRA config and training hyperparameters, and run it in one line. It supports Llama, Mistral, Qwen, Gemma, Phi… loads. It keeps compatibility with the HuggingFace Hub to download models and datasets, and with MLflow / W&B for tracking. It is the convenience framework this blog cites when it talks about “productive fine-tuning without reinventing the wheel”. Gotcha: the pace of change in the community is fast; a config.yml that worked six months ago can break with a current version because of internal refactors. Pinning the exact Axolotl version in the environment mitigates this. Licence Apache 2.0, maintained by the OpenAccess AI Collective (community-driven). A very similar alternative, more widely used in China: LLaMA-Factory (Apache 2.0, Beihang U.).

Ray Train

Ray Train scales fine-tuning to multiple nodes by distributing workers across a Ray cluster. While DeepSpeed and FSDP are intra-job parallelism (several GPUs collaborating on one job), Ray Train is the orchestration plane that builds the cluster, launches workers, manages checkpoints, recovers from node failures, and integrates with Slurm or Kubernetes. For training runs beyond 8 GPUs on changing clusters, Ray Train avoids the operational burden of “manually launching N torchrun processes with NCCL”. It combines with MLflow for tracking. Gotcha: the Ray learning curve is real; for a single node with 4-8 GPUs, torchrun or Hugging Face Accelerate are simpler. Ray Train shines when there are N changing nodes. Licence Apache 2.0, maintained by Anyscale Inc. (commercial backer) + community. A more K8s-native alternative: Kubeflow Training Operator (Apache 2.0, LF AI & Data).

More options for Tune:

DeepSpeed — ZeRO parallelism in 3 stages, mixed precision, CPU/NVMe offload. MIT, Microsoft.
FSDP (Fully Sharded Data Parallel) — native PyTorch parallelism, an alternative to DeepSpeed. BSD, Meta.
LLaMA-Factory — equivalent to Axolotl with a focus on the Llama family. Apache 2.0, Beihang University.

Decision matrix — Tune:

If your case is	Choose
Fine-tune on 1 GPU with 24GB (RTX 4090)	QLoRA with bitsandbytes NF4 + Axolotl
Fine-tune on 1 H100 80GB, models < 13B	LoRA bf16 + Axolotl
Fine-tune on 4-8 GPUs, single node	transformers + PEFT + Accelerate + MLflow
Multi-node fine-tune on a K8s cluster	Kubeflow Training Operator or Ray Train
Reproducible tracking is mandatory	MLflow + DVC input artefact
You want the minimum viable	Axolotl + MLflow

Stage 3 — Eval + Guardrails

Eval validates candidates pre and post promotion against a golden set with operational metrics; Guardrails runs safety online. Detailed in the posts on evals and guardrails.

DeepEval

DeepEval is the “pytest-style” OSS eval suite: you define tests with assertions over faithfulness, answer relevancy, contextual precision, hallucination rate, summarisation quality… and run them in CI. Each metric is an evaluator: some rule-based, others LLM-as-judge with auditable prompts. The philosophy is “evals as unit tests”: parameterisable by dataset, failable in CI, integrable with GitHub Actions. Gotcha: LLM-as-judge metrics vary between judge model versions, so if the judge moves up a version, the thresholds lose their earlier statistical meaning. Explicit pinning of the judge model in config plus periodic threshold recalibration is mandatory discipline. Licence Apache 2.0, maintained by Confident AI (a company); parallel commercial SaaS offering. Comparable: TruLens (MIT, TruEra) and G-Eval (academic).

RAGAS (RAG Assessment)

RAGAS specialises in evaluating RAG pipelines. It defines four canonical metrics: faithfulness (the answer is supported by the retrieved chunks), answer relevancy (the answer addresses the query), context precision (the retrieved chunks are relevant), context recall (all the relevant chunks were retrieved). Each metric is computed with LLM-as-judge over a dataset of (query, context, expected answer). For a RAG system, RAGAS is the evaluator that measures whether retrieval is aligned with generation. It integrates with Langfuse and MLflow to store results. Gotcha: RAGAS works well with golden sets of under 1000 examples; over huge golden sets the cost of the LLM judge per evaluation explodes, and the practice is to sample. Licence Apache 2.0, maintained by Exploding Gradients (the authors’ company).

Promptfoo

Promptfoo is the declarative, CI-oriented evaluator: in promptfooconfig.yaml you define a set of prompts and a set of assertions (contains text X, does not contain Y, faithfulness > 0.8, judge approves…), point at a provider (OpenAI compatible, vLLM, Ollama…), and promptfoo eval runs the matrix of prompts × providers × assertions, returns a diff against the baseline and fails CI if anything regresses. It is the most “DevOps-friendly” piece of the evals ecosystem: it integrates trivially with GitHub Actions, GitLab CI or Jenkins. Gotcha: assertion thresholds have to be calibrated with real data; starting at a default > 0.5 produces false positives that erode the team’s trust. Calibrate after the first week. Licence MIT, maintained by Promptfoo, Inc. (a company); a commercial Promptfoo Cloud SaaS offering exists but the OSS version is complete.

NeMo Guardrails

NeMo Guardrails is NVIDIA’s framework for defining and enforcing policies in LLM systems through a DSL called Colang. It lets you express rules such as “if the user asks about topic X, answer with template Y” or “if the model tries to do Z, block it” in a conversational-script syntax rather than in Python. It runs as middleware between the app and the model: input rails (validate what comes in), output rails (validate what goes out), dialog rails (control the flow). Designed for complex multi-turn systems where the policies are nontrivial. Gotcha: Colang adds latency per turn (roughly 50-200 ms depending on the policy graph); for high-throughput conversational chat, dialog rails are disabled and only input + output remain. Licence Apache 2.0, maintained by NVIDIA.

Microsoft Presidio

Presidio is the most mature OSS PII (Personally Identifiable Information) detector in the ecosystem. It detects DNI, NIE, IBAN, phone numbers, emails, physical addresses, credit card numbers, personal names, dates of birth… with recognisers based on regex + NER (spaCy) + custom validators. It supports redaction (replace with placeholders), masking (asterisks) or deterministic anonymisation (repeatable hash). For ENS/NIS2 scenarios, it is the piece placed in front (on input) and behind (on output) the LLM to guarantee that no PII is processed or emitted. Gotcha: the built-in recognisers cover English well and the rest badly; for Spanish, Catalan and Basque you have to add custom recognisers, which is disciplined work but doable. Licence MIT, maintained by Microsoft.

More options for Eval:

Phoenix Arize OSS — combines tracing + evals, an alternative to Langfuse Evals. ELv2, Arize AI.
lm-eval-harness — academic suite with standard benchmarks (MMLU, HellaSwag…). MIT, EleutherAI.
HELM — holistic academic evals. Apache 2.0, Stanford CRFM.
Guardrails AI — a pythonic alternative to NeMo Guardrails. Apache 2.0, Guardrails AI Inc..
LlamaGuard / PromptGuard / ShieldGemma — safety models, not frameworks. Open weights, Meta / Google.

Decision matrix — Eval + Guardrails:

If your case is	Choose
Eval in CI, “pytest for LLMs” style	Promptfoo + GitHub Actions
Eval specific to a RAG pipeline	RAGAS + Langfuse datasets
General eval with custom metrics	DeepEval + MLflow dataset
Dialog policy with declarative rules	NeMo Guardrails (Colang)
PII redaction in/out only	Presidio (you do not need NeMo)
Open safety model in Spanish	LlamaGuard 3 or ShieldGemma

Stage 4 — Deploy

Deploy serves tokens to the user with predictable throughput and latency, adapter hot-swap and multi-tenancy where it applies. Covered in the posts on vLLM on K8s, LLM operators, multi-tenant cluster, KV cache, PagedAttention and disaggregated serving.

vLLM

vLLM is the reference OSS inference engine. It implements PagedAttention (KV cache paging in the style of virtual memory, avoiding fragmentation), continuous batching (requests join the batch as they arrive, rather than waiting for the next batch), prefix caching (common prefixes, system prompts, do not recompute the KV cache), LoRA hot-swap (--enable-lora allows loading and unloading adapters without restarting the engine), an OpenAI-compatible API, and disaggregated prefill/decode support since 2025. It covers almost everything from Llama 3 / Mistral / Qwen / DeepSeek. Gotcha: maximum throughput is only reached with --max-num-seqs and --gpu-memory-utilization tuned for the specific model and hardware; the defaults are conservative. The initial tuning session pays off: 2-3× throughput. Licence Apache 2.0, originated at UC Berkeley, today maintained by the vLLM Project / LF AI & Data plus a broad community (Red Hat, NVIDIA, AWS, IBM contribute). Serious alternatives in the same bucket: TGI (Apache 2.0, Hugging Face), SGLang (Apache 2.0, LMSys), TensorRT-LLM (Apache 2.0, NVIDIA, requires conversion).

KServe

KServe is the Kubernetes operator for serving ML models, LLMs included, in a declarative pattern: you define an InferenceService YAML with the model and predictor (which can be vLLM, TGI, Triton, or a custom container) and KServe handles scheduling onto GPU nodes, autoscaling (scale-to-zero included), traffic splitting for canary, and model registry integration. It is the layer that standardises “how a model is deployed on K8s” across multiple engines, instead of inventing engine-specific YAML. It supports multi-model with Inference Graphs (chaining preprocessor → model → postprocessor) and integrates with KEDA/Karpenter for GPU pool autoscaling. Gotcha: scale-to-zero on GPU works badly in practice because the warm-up (loading weights into VRAM) takes tens of seconds; minReplicas: 1 is better. Licence Apache 2.0, maintained by Kubeflow / LF AI & Data. Alternatives: KubeRay (Apache 2.0, Anyscale), llm-d (Apache 2.0, CNCF), KAITO (MIT, Microsoft Azure).

Triton Inference Server

Triton serves heterogeneous models on a single backend: LLMs (via the vLLM or TensorRT-LLM backend), traditional models (ONNX, TorchScript, TensorFlow), custom models. For systems that mix LLM inference with traditional classifiers, embedding encoders, reranking models, OCR and so on, Triton avoids having N different engines in N pods. It supports ensemble models (chaining models in a single request), dynamic batching, model versioning, model warmup. Gotcha: Triton is flexible but heavy to operate; for systems serving only LLMs, vLLM directly is simpler and more optimised. Triton shines when there is real heterogeneity. Licence BSD-3-Clause, maintained by NVIDIA.

Envoy AI Gateway

Envoy AI Gateway is the CNCF ecosystem’s “LLM-aware API gateway”. Built on Envoy Proxy, it adds knowledge of the OpenAI-compatible APIs (chat completions, embeddings, etc.), routing across multiple backends (local vLLM + OpenAI + Anthropic + Bedrock), token-based rate limiting (limits by tokens per minute, not by requests), intelligent retries, fallback between providers, and built-in OTel observability. It is the piece that makes “AI Gateway” real as an architectural category. Gotcha: integration with authentication (OIDC, JWT) is flexible but requires detailed Envoy configuration; an “out of the box” AI Gateway with no configuration produces an Envoy that passes everything through. Licence Apache 2.0, maintained by CNCF since the initial donation from Tetrate. Alternatives: LiteLLM Proxy (MIT, BerriAI), Portkey (MIT, Portkey AI), Kong AI Gateway (Apache 2.0 base + EE, Kong Inc.).

llama.cpp

llama.cpp serves LLMs on CPUs (and Apple Silicon, GPUs via Vulkan/Metal/CUDA) with very aggressive quantisation (GGUF format, down to 2-bit). It is the canonical option for inference on hardware without a dedicated GPU: edge devices, workstations, development machines. It covers everything from small models (Phi-3, Gemma 2B) to Llama 70B on hardware with enough RAM. Gotcha: latency on CPU is orders of magnitude worse than on a dedicated GPU, so it is useful for offline evals, drift checks and local development, not for production serving under real load. Licence MIT, maintained by Georgi Gerganov + community.

More options for Deploy:

TensorRT-LLM — maximum optimisation on NVIDIA Hopper/Ada. Apache 2.0, NVIDIA.
SGLang — good for workloads with structured generation and JSON. Apache 2.0, LMSys.
TGI — a mature alternative, focused on the HuggingFace ecosystem. Apache 2.0, HuggingFace.
NVIDIA Dynamo — multi-node disaggregated serving. Apache 2.0, NVIDIA.
llm-d — K8s operator specific to LLMs. Apache 2.0, CNCF.

Decision matrix — Deploy:

If your case is	Choose
Production serving on NVIDIA H100/A100	vLLM (the safe default)
Absolute throughput squeezing on Hopper	TensorRT-LLM + vLLM plugin or standalone
Edge / local dev without GPU	llama.cpp
Multi-model (LLM + classifiers + encoders)	Triton with the vLLM backend
Declarative K8s with autoscaling	KServe + vLLM as predictor
AI Gateway with token rate limiting	Envoy AI Gateway
Multi-node disaggregated GPU cluster	NVIDIA Dynamo over vLLM

Stage 5 — Observe

Observe propagates trace_id end-to-end, emits runtime metrics, runs an LLM judge over sampling and detects drift. Detailed in tracing with AgentSight, MCP observability with OTel and eBPF + drift.

OpenTelemetry Collector

The OTel Collector is the agent that receives traces, metrics and logs in OTel format (or in any other via receivers), processes them (filters, sampling, attribute enrichment, PII redaction), and routes them to one or several backends (Tempo, Jaeger, Prometheus, Loki, Langfuse…). It is the piece that decouples the apps from the observability backend: moving from Tempo to Jaeger means changing the Collector’s exporter, not the app. For LLMOps it matters especially because the OTel GenAI semantic conventions spec defines the attributes gen_ai.request.model, gen_ai.prompt.version, gen_ai.response.tokens and so on, which stitch the trace_id to the system’s lineage. Gotcha: the Collector configuration tends to grow; without discipline and periodic review it ends up as an 800-line YAML that nobody understands. Modularising with extensions helps. Licence Apache 2.0, maintained by the CNCF / OpenTelemetry Project.

Tempo (traces) + Jaeger

Grafana Tempo is the distributed trace backend optimised for cost: it uses an object store (S3/MinIO) instead of Elasticsearch, deduplicates by trace_id, and integrates natively with Grafana for visualisation. For LLMOps, where a real request generates 10-30 spans (gateway, prompt pull, RAG retrieval, prefill, decode N times, scoring), Tempo copes with high volumes at a reasonable cost. Jaeger is the more established CNCF alternative, better for cases under 100k traces per day, worse for native object store. Gotcha: Tempo has no traditional indexing; searches like “traces that took more than 5s and touched tenant X” require TraceQL + Grafana and are not as fast as Jaeger with Elasticsearch. For immediate ad-hoc diagnosis it is worth keeping a parallel Jaeger with aggressive sampling. Licences AGPL 3.0 (Tempo) and Apache 2.0 (Jaeger), maintained by Grafana Labs and CNCF respectively.

Prometheus + Grafana

Prometheus is the time-series metrics foundation of the ecosystem. Pull model (it scrapes /metrics endpoints), PromQL for queries, exporters for everything (Postgres, Kafka, NVIDIA GPU via dcgm-exporter, vLLM natively). Grafana visualises Prometheus + Tempo + Loki on a single plane. For LLMOps, the critical metrics are gpu_utilization, kv_cache_usage_pct, tokens_per_second, prefill_latency_p95, decode_latency_p95, queue_depth, aggregated by tenant. Gotcha: Prometheus is very good up to about 1M active series; above that it is worth moving to Thanos or Mimir for long retention and horizontal scalability. For the blog’s typical LLM cluster (4-8 H100), Prometheus alone is enough. Licences Apache 2.0 (Prometheus, CNCF) and AGPL 3.0 (Grafana 10+, Grafana Labs).

Langfuse

Langfuse is the LLM-specific OSS observability + prompt management tool. It captures spans with LLM semantic conventions (input, output, model, tokens, latency, score, user_id, session_id), visualises them as conversational traces (not just span trees), manages prompts versioned with a production label and allows curated datasets + evals from the same UI. For serious LLMOps, Langfuse fills the gap that neither Tempo nor Jaeger covers: a tracing UI designed to be LLM-first. Gotcha: Langfuse maintains its own store (Postgres + ClickHouse for high volume); on large clusters the operational side of ClickHouse deserves attention. To get started, Postgres alone copes. Licence MIT for the OSS core, EE Enterprise Edition with additional features (SSO, audit logs, advanced RBAC). Maintained by Langfuse GmbH (Berlin, German). There is Langfuse Cloud (SaaS).

Phoenix Arize OSS

Phoenix is Arize AI’s OSS for LLM observability + evals, an alternative to Langfuse with a different emphasis: more oriented to evaluation and visual debugging (embedding drift, cluster analysis), less to prompt management. A good pairing with Langfuse when you want a dual approach: Langfuse for “production conversational traces”, Phoenix for “exploratory investigation of model behaviour”. Gotcha: Phoenix duplicates functionality with Langfuse and with MLflow; having all three in production multiplies the operational burden. Pick one as the main tool and the others as complements. Licence Elastic License 2.0 (not strictly OSI), maintained by Arize AI.

Cilium Tetragon + Hubble

Tetragon (eBPF runtime security observer) and Hubble (eBPF network observer) are the low-level pieces that give real runtime visibility into the cluster: which processes run in which pods, what syscalls they make, what network connections they open, in real time. For ENS/NIS2 environments that demand “prove what ran in production”, Tetragon is the irrefutable audit layer: every process execution with its parent, its capabilities, its K8s context. Hubble visualises network flows by pod, namespace, service. Gotcha: the volume of events generated is high; without kernel-side filtering (which Tetragon supports with TracingPolicy), it saturates the observability plane fast. Discipline in policies. Licence Apache 2.0 for both, maintained by Cilium / CNCF / Isovalent.

Evidently AI

Evidently is the OSS library for drift detection: it compares input and output distributions between two time windows (training vs production, current week vs previous week), applies statistical tests (KS, PSI, Wasserstein, chi-square) and generates HTML reports. For LLMOps it detects when the distribution of prompts changes (new topics, new lengths, new languages) or when the model starts answering shorter / longer / differently. Gotcha: Evidently is oriented to tabular data and embeddings; for raw text it is best combined with an encoder embedder that produces vectors before applying the tests. Licence Apache 2.0, maintained by Evidently AI (a company). Alternatives: NannyML (Apache 2.0, NannyML BV), Alibi Detect (Apache 2.0, Seldon).

More options for Observe:

Loki — Prometheus-style logs backend for Grafana. AGPL 3.0, Grafana Labs.
Pixie — auto-instrumented eBPF observability. Apache 2.0, CNCF.

Decision matrix — Observe:

If your case is	Choose
Minimum viable stack	OTel Collector + Tempo + Prometheus + Grafana + Langfuse
Traces with strong ad-hoc search	Add Jaeger with aggressive sampling
ENS / NIS2 compliance runtime audit	Tetragon + Hubble + mandated retention
Exploratory investigation of the model	Phoenix Arize OSS alongside Langfuse
Statistical drift detection	Evidently over embeddings + inputs
Cluster beyond 1M Prometheus series	Mimir (Grafana Labs) or Thanos

Stage 6 — Retrain + cross-cutting pieces

Retrain closes the loop feedback → triage → enriched dataset → new adapter. Prompt versioning and data versioning stitch lineage across stages. Detailed in retrain, prompt versioning and data versioning.

Apache Airflow

Airflow is the most established OSS DAG scheduler. You define workflows as Python code (DAGs), each DAG with tasks (operators) that run according to declared dependencies plus a cron schedule. For retraining: a weekly DAG that extracts feedback from Postgres, triages it with an LLM-as-classifier, enriches the enriched dataset in DVC, launches the fine-tuning job on Kubernetes, runs evals against the golden set, and promotes if it passes the gates. There is a huge ecosystem of operators for everything (S3, Postgres, Kafka, Slack, K8s, Spark…). Gotcha: Airflow 2.x improved a great deal over the chaos of 1.x, but the scheduler is still a component that deserves operational attention (Postgres backend, executor pool, sidecar workers); for simple pipelines it is over-engineering. Licence Apache 2.0, maintained by the ASF.

Argo Workflows

Argo Workflows is the K8s-native equivalent of Airflow: each step is a container, the DAGs are defined as K8s YAML, the executor is Kubernetes itself. For environments where everything is K8s, Argo fits without an extra component to maintain. Long tasks (a six-hour fine-tuning run) execute as Pods that survive control plane failures. It integrates trivially with Kubeflow Pipelines (which is built on top). Gotcha: Argo’s YAML syntax is verbose; for complex DAGs, Argo feels less productive than Airflow in Python. Solutions: Hera (Python DSL for Argo, a DataBricks contribution) or Argo + custom CRDs. Licence Apache 2.0, maintained by CNCF.

Kubeflow Pipelines

Kubeflow Pipelines is the layer above Argo Workflows oriented specifically to ML: artefact tracking, experiment tracking, reusable pipeline templates, versioned components. Built on Argo, it adds the ML conceptual model (input artefact, output artefact, metrics) that raw Argo does not have. For cyclical retraining on a K8s cluster, it is the most “ML-ready” option in the OSS ecosystem. Gotcha: Kubeflow as a full suite is heavy (10+ components); many organisations install only Pipelines + Training Operator + Katib and skip Notebook Server / legacy KFServing. Licence Apache 2.0, maintained by CNCF / LF AI & Data.

Feast

Feast is the most widely used OSS feature store. It defines feature views over batch sources (BigQuery, Postgres, Parquet) and online sources (Redis, DynamoDB, Postgres with an extension), exposes a consistent API for read-during-training and read-during-inference (point-in-time correctness), and guarantees that the features of the model in production are the same ones it was trained with. For LLMOps where the model needs consistent user / session / context features (latest plan, tenure as a customer, recent tickets), Feast provides the discipline. Gotcha: for many pure LLM systems (a RAG chatbot with no complex features), Feast is over-engineering, Postgres is enough. When there are real features (recommendation, scoring, ranking), Feast shines. Licence Apache 2.0, maintained by LF AI & Data.

Argilla

Argilla is the OSS annotation + HiL (human-in-the-loop) platform most aligned with modern LLMOps. It creates annotation projects with templates (classification, ranking, span annotation, RLHF preference, free-form text), connects to HuggingFace datasets, and integrates with Langfuse to import traces from production as cases to annotate. It supports multiple annotators with reconciliation, kappa scoring and quality control. To enrich retrain datasets with cases from the “blunt tone” cluster of the Retrain post, Argilla is the front end. Gotcha: Argilla requires Elasticsearch for production performance; for small experiments SQLite is enough. Licence Apache 2.0, maintained by Argilla, Inc. (acquired by Hugging Face in 2024). Alternative: Label Studio (Apache 2.0, HumanSignal), more generalist, less LLM-first.

Langfuse Prompts + MLflow Prompt Registry

Langfuse Prompts manages prompts as versioned entities with labels (production, staging, experiment). The client reads the active prompt from Langfuse on the request path (with a local cache of a few seconds) and propagates prompt_id, prompt_version to the OTel span, exactly as the forensic post does. MLflow Prompt Registry does the same with a slightly different conceptual model (no labels-as-pointers; it uses stages, like the Models registry). Both are valid; the choice depends on which tracking tool is already in place. Gotcha (Langfuse): labels are mutable, so changing production points at another version with no explicit audit; prompts are better deployed via PR against the configs repo, not manually in the UI. Licences and governance covered above.

More options for Retrain + cross-cutting pieces:

Prefect — “modern” Python DAGs, an alternative to Airflow. Apache 2.0, Prefect Tech.
Dagster — DAGs with a strong focus on data assets. Apache 2.0, Dagster Labs.
Label Studio — generalist annotation. Apache 2.0, HumanSignal.
OpenLineage — cross-system lineage event standard. Apache 2.0, LF AI & Data.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage with a UI. Apache 2.0.

Decision matrix — Retrain + cross-cutting pieces:

If your case is	Choose
Simple pipelines with a catalogue of operators	Airflow
Everything is K8s, minimal components	Argo Workflows
ML pipelines with artefact tracking	Kubeflow Pipelines
HiL annotation for LLM retraining	Argilla + Langfuse integration
Features shared between training and inference	Feast
No complex features, only prompts + LLM	Skip Feast
Lightweight prompt registry	Langfuse Prompts
MLflow is already centralised	MLflow Prompt Registry

Master table: licence, governance and enterprise offering

Tool	Licence	Governance / maintainer	Commercial EE / SaaS
DVC	Apache 2.0	Iterative.ai	DVC Studio
lakeFS	Apache 2.0	Treeverse	lakeFS Cloud
MinIO	AGPL v3	MinIO Inc.	SUBNET / AIStor
Qdrant	Apache 2.0	Qdrant GmbH	Qdrant Cloud
pgvector	PostgreSQL License	Andrew Kane + community	— (built-in Postgres clouds)
PostgreSQL	PostgreSQL License	PostgreSQL Global Dev Group	several managed (Crunchy, Neon, Aiven, EDB)
Apache Kafka	Apache 2.0	ASF	Confluent Cloud
Debezium	Apache 2.0	Red Hat / ASF	Debezium Server / Confluent Connectors
Apache Flink	Apache 2.0	ASF	Ververica Platform, Aiven
HF Transformers	Apache 2.0	Hugging Face SAS	HF Inference Endpoints / Enterprise Hub
PEFT	Apache 2.0	Hugging Face SAS	— (part of the HF offering)
bitsandbytes	MIT	Tim Dettmers + community	—
MLflow	Apache 2.0	LF AI & Data	Databricks MLflow
Axolotl	Apache 2.0	OpenAccess AI Collective	—
Ray (Train)	Apache 2.0	Anyscale + community	Anyscale Platform
DeepSpeed	MIT	Microsoft	—
DeepEval	Apache 2.0	Confident AI	Confident AI SaaS
RAGAS	Apache 2.0	Exploding Gradients	—
Promptfoo	MIT	Promptfoo, Inc.	Promptfoo Cloud
NeMo Guardrails	Apache 2.0	NVIDIA	NeMo Microservices
Presidio	MIT	Microsoft	—
Phoenix (Arize)	Elastic v2	Arize AI	Arize Platform
vLLM	Apache 2.0	vLLM Project / LF AI & Data	several (Red Hat, AWS, IBM, NVIDIA)
TGI	Apache 2.0	Hugging Face SAS	HF Inference Endpoints
SGLang	Apache 2.0	LMSys + community	—
TensorRT-LLM	Apache 2.0	NVIDIA	NVIDIA AI Enterprise
llama.cpp	MIT	Georgi Gerganov + community	—
Triton Inference Server	BSD-3	NVIDIA	NVIDIA AI Enterprise
KServe	Apache 2.0	LF AI & Data (Kubeflow)	—
Envoy AI Gateway	Apache 2.0	CNCF / Tetrate	Tetrate Service Bridge
LiteLLM	MIT	BerriAI	LiteLLM Cloud
OpenTelemetry	Apache 2.0	CNCF	several vendors (Honeycomb, Datadog, Grafana)
Tempo	AGPL 3.0	Grafana Labs	Grafana Cloud Tempo
Jaeger	Apache 2.0	CNCF	—
Prometheus	Apache 2.0	CNCF	Grafana Cloud, AMP, GCP Managed Prom, Azure
Grafana	AGPL 3.0	Grafana Labs	Grafana Cloud, Grafana Enterprise
Loki	AGPL 3.0	Grafana Labs	Grafana Cloud Loki
Langfuse	MIT (core) / EE	Langfuse GmbH	Langfuse Cloud
Tetragon	Apache 2.0	Cilium / CNCF / Isovalent	Isovalent Enterprise
Hubble	Apache 2.0	Cilium / CNCF	Isovalent Enterprise
Evidently AI	Apache 2.0	Evidently AI	Evidently Cloud
Apache Airflow	Apache 2.0	ASF	Astronomer, MWAA, Cloud Composer
Argo Workflows	Apache 2.0	CNCF	—
Kubeflow Pipelines	Apache 2.0	CNCF / LF AI & Data	—
Feast	Apache 2.0	LF AI & Data	Tecton (commercial)
Argilla	Apache 2.0	Hugging Face	HF Hub features
OpenLineage	Apache 2.0	LF AI & Data	—
DataHub	Apache 2.0	Acryl Data	Acryl Cloud

The pattern to watch for when reading the table: AGPL 3.0 and Elastic v2 are the ones that add the most friction in companies with strict licensing policies (legal asks for a specific review). The Apache 2.0 ones pass compliance without argument. Those with an “EE Enterprise” tier or equivalent hide a decision: the OSS version is functionally complete for production, but team features (SSO, audit, advanced RBAC) live in the commercial version. For ENS customers under a HIGH declaration, the EE features (SSO with corporate SAML/OIDC, immutable audit logs) are usually mandatory, so it is worth knowing the price beforehand.

When to move from the “minimum stack” to the “full stack”

The whole catalogue can be intimidating. But you do not build all of it on day one. There is a reasonable order that this blog has been validating in posts throughout the series. The minimum viable stack that serves an LLM API with acceptable discipline:

Serving: vLLM on Kubernetes plus an Envoy AI Gateway in front.
Data: Postgres + pgvector (no Qdrant), MinIO for object store, no Kafka.
Tune: Axolotl + MLflow, no Ray Train.
Eval: Promptfoo in CI, no RAGAS and no judge in production.
Observe: OTel Collector + Prometheus + Grafana + Langfuse, no Phoenix and no Tetragon.
Retrain: feedback in Postgres + crontab scripts, no Airflow.
Versioning: prompts in Langfuse + datasets in DVC over MinIO, no lakeFS.

That is about 8-10 components and it serves a reasonable LLM system for a single tenant with moderate traffic. As the system grows, there are identifiable moments where adding each piece pays off:

Trigger	Component to add
Multi-tenant with isolated corpora	Qdrant (collections per tenant, ACL)
The corpus is refreshed often and periodically breaks	lakeFS (branches with pre-merge hooks)
The embedding pipeline needs streaming	Kafka + Debezium + Flink
Retraining moves from monthly to weekly	Airflow or Argo Workflows
Shared features appear (customer profile, scoring)	Feast
Annotation outgrows informal capacity	Argilla
RAG eval needs specific metrics	RAGAS + Langfuse datasets
ENS compliance demands runtime audit	Tetragon + Hubble
Drift is invisible and shows up late	Evidently
A single stack stops covering multi-model	Triton or KServe with several predictors
Multiple simultaneous multi-tenant adapters	vLLM Production Stack + a dedicated Operator

Each jump adds 1-2 components and is worth the cost only when the trigger is clear. Adding Kafka “just in case” when the corpus is updated once a month is net negative work.

What we have not covered (yet)

Some pieces still deserve a post of their own:

Schema Registry for LLM data and prompts (Confluent OSS, Karapace, JSON Schema Registry).
Catalogue + lineage in depth: DataHub vs Atlas vs OpenMetadata + OpenLineage taken seriously.
Federated learning over OSS (Flower, FedML) for scenarios where the data is not centralised.
OSS MCP Servers and their place in the stack as a tools / actions layer.
“Agentic” evals specific to multi-step systems with tool use.
Upgrade best practices for each component (vLLM every 6 weeks, Kafka major every 18 months, etc.).

References

vLLM · vLLM Production Stack · TGI · SGLang · TensorRT-LLM · llama.cpp — OSS inference engines.
Triton Inference Server · KServe · Envoy AI Gateway · LiteLLM — orchestration and AI gateway.
Qdrant · pgvector · Milvus — vector databases.
DVC · lakeFS · MinIO — versioning and object store.
Apache Kafka · Debezium · Apache Flink — streams and CDC.
Hugging Face Transformers · PEFT · bitsandbytes · Axolotl — fine-tuning.
MLflow · Ray Train · Kubeflow Training Operator — training orchestration.
DeepEval · RAGAS · Promptfoo · NeMo Guardrails · Presidio — evals and guardrails.
OpenTelemetry · OTel GenAI semconv · Tempo · Prometheus · Grafana · Loki — observability foundation.
Langfuse · Phoenix Arize · Evidently AI — LLM observability and drift.
Cilium · Tetragon · Hubble — eBPF runtime.
Apache Airflow · Argo Workflows · Kubeflow Pipelines · Feast · Argilla — orchestration + retrain + annotation.

The parallel catalogue: the six LLMOps stages in open source and on the hyperscalers (AWS, GCP, Azure)

Sat, 23 May 2026 07:00:00 +0200

TL;DR

The previous forensic post used a single request to walk the six stages of the LLMOps pipeline and the two cross-cutting components. This post walks the same stages but crosses them with three extra columns: how each stage is built in open source on-premise, and what the equivalent services are on AWS, GCP and Azure. It is not a migration guide nor a cost benchmark: it is a catalogue of equivalences with its gaps. The general pattern you will see: OSS gives you control, sovereignty and free composition in exchange for expensive operations; the hyperscalers give you integration and time-to-market in exchange for lock-in on margins, data contracts and political dependency. For scenarios subject to ENS / NIS2 with critical customer data, OSS wins by default; for discovery projects where time-to-market is the deciding metric, the hyperscaler wins by default. The interesting part is in the middle. As a concrete thread, at the end we take the multi-tenant chatbot from the previous post and port it to AWS piece by piece to show what disappears, what appears, and where the lock-in materialises.

You are here: the same six stages, but by column

This post shares a map with the previous one, the six stages and the two cross-cutting pieces are all active, but it changes the cut: instead of following a request horizontally, it makes the vertical cut and shows which tools live in each stage depending on the deployment model.

The analogy: your own bakery and the franchise

A baker opens a business. There are two possible models.

He can open his own bakery: rent the premises, buy the oven, choose the flour suppliers, hire his master baker, write his recipes, set the prices, dress the shop window. The day he wants to launch an organic rye sourdough loaf, he asks nobody for permission. The day the price of flour goes up, he finds another supplier. But he pays for all of it: the initial investment, the risk, the daily operations, the months when he misreads the neighbourhood. The bakery is his.

Or he can join a franchise: the franchisor hands over the turnkey premises, the oven with a maintenance contract, the suppliers already negotiated, the operating manuals, the written recipes, the centralised marketing, the loyalty app, the POS system. The learning curve is weeks, not years. But the recipes belong to the franchisor, so do the suppliers, the price of bread is in the catalogue, and the day the croissant formula changes he gets an informational email, not a business decision.

Both bakeries turn out bread. Both meet health regulations and generate income. The operational difference is enormous and it is not about technology: it is about ownership, control and timescale.

The parallel with LLMOps is direct. The on-premise OSS stack is the bakery you own. The managed stack on the hyperscalers is the franchise. The pieces that appear in each stage are functionally equivalent, at the end of the day both solve the same technical problem, but the governance model, the operational cost, the lock-in and the compliance guarantees are different. This post builds the parallel catalogue so that the choice is not made by default.

Quick recap of the previous post

In the forensic post we followed one specific request: a premium-es user of an insurance company asking “How do I cancel my premium subscription?” to the multi-tenant support chatbot of the SaaS provider that hosts it. The route crossed the six stages of the LLMOps pipeline, Data, Tune, Eval, Deploy, Observe, Retrain, plus the two cross-cutting components, prompt versioning and data versioning, over an on-premise infrastructure: RKE2 with Cilium BGP, a 4×H100 SXM cluster, an RTX 4090 for development, vLLM on Kubernetes, Langfuse + OTel + Prometheus + Tempo, Postgres + Qdrant, DVC + lakeFS + MinIO, Kafka and MLflow. The system meets ENS / NIS2 and keeps trace_id propagated end to end.

What comes now is that same system, piece by piece, showing for each box which tool does the job if you are on public cloud, because the integrator’s question is rarely “OSS yes or no?”: it is “what do I lose and what do I gain if I take this box managed?”. And the answer differs box by box.

Stage 1 — Data

The problem. There are three sub-problems the Data stage solves, frequently confused with one another. First, versioning and identity of the corpus and of the training datasets (making sure a dataset_id, dataset_version exists and propagates). Second, storage and serving of the operational corpus (object store + vector index + structured text). Third, streams and ingestion from source systems with CDC, transformation and stable schemas (Schema Registry).

Reference OSS stack. Versioning lives in DVC (pointers in git, content in an object store) combined with lakeFS for branch/merge semantics over data. The post on data versioning goes into the functional difference. The object store is MinIO or Ceph. The vector index is Qdrant or Milvus for large corpora (millions of chunks) and pgvector on Postgres 18 for small cases where having one component fewer pays off operationally. The stream layer is Kafka (pure Apache or Redpanda) with a Schema Registry (Confluent or Karapace OSS), CDC with Debezium or Flink CDC, transformation with Flink or Spark Structured Streaming. The catalogue / lineage is DataHub, Apache Atlas or OpenMetadata with OpenLineage events between systems. The post on PostgreSQL + Qdrant ingestion and the post on RAG over Kafka cover the detailed operations.

Hyperscaler equivalents. On AWS, the corpus lives in S3 (with versioning enabled, which is the cheap substitute for serious data versioning), tabular queries in Athena or Redshift, the vector index in Amazon OpenSearch with the vector plug-in or in Amazon Aurora pgvector. The stream layer is MSK (managed Kafka) or Kinesis Data Streams, CDC with AWS DMS, transformation with Glue Streaming or MSK Connect. The catalogue is AWS Glue Data Catalog + AWS Lake Formation for data governance. And for the RAG case there is also Amazon Bedrock Knowledge Bases, the managed shortcut: you give it S3, it indexes into OpenSearch or Aurora pgvector, exposes a retrieval API and the operations end there, in exchange for paying per indexed chunk and per query.

On GCP, the corpus lives in Cloud Storage (with object versioning), the analytical warehouse is BigQuery (with BigQuery Vector Search already integrated), the dedicated vector store is Vertex AI Vector Search (formerly Matching Engine). The stream layer is Pub/Sub + Dataflow, CDC with Datastream. Catalogue and lineage are Dataplex (which in 2024-2025 absorbed Data Catalog and added automatic lineage). The managed equivalent of Knowledge Bases is Vertex AI Search (formerly Discovery Engine).

On Azure, the corpus lives in ADLS Gen2, tabular queries in Microsoft Fabric / Azure Synapse, the vector index in Azure AI Search (vector mode) or Azure Cosmos DB for PostgreSQL with pgvector. The stream layer is Event Hubs + Stream Analytics or Microsoft Fabric Real-Time Intelligence, CDC with Azure Data Factory. The catalogue is Microsoft Purview, which covers catalogue, lineage and data governance integrated with Entra ID.

Summary table — Data stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Object store	MinIO, Ceph	S3	Cloud Storage	ADLS Gen2
Dataset versioning	DVC, lakeFS	S3 Versioning (limited), Lake Formation	GCS Versioning, Dataplex	ADLS versioning, Purview
Vector index	Qdrant, Milvus, pgvector	OpenSearch, Aurora pgvector, Bedrock KB	Vertex Vector Search, BigQuery VS	Azure AI Search, Cosmos pgvector
Stream + CDC	Kafka + Debezium + Flink	MSK / Kinesis + DMS + Glue	Pub/Sub + Datastream + Dataflow	Event Hubs + ADF
Schema Registry	Karapace, Confluent OSS	Glue Schema Registry	Pub/Sub schemas	Schema Registry (Event Hubs)
Catalogue + lineage	DataHub, Atlas, OpenLineage	Glue Catalog + Lake Formation	Dataplex	Purview
Managed end-to-end RAG	— (you build it)	Bedrock Knowledge Bases	Vertex AI Search	Azure AI Studio Knowledge

Where the names mislead. S3 Versioning is not DVC. It keeps object versions but has no notion of a dataset (which objects together make up version 3 of the enriched set?), does not propagate dataset_hash to the trainer, does not integrate with experiment tracking, and does not fail a CI run if a dataset breaks schema. Covering that properly on AWS requires combining it with Lake Formation, Glue Data Catalog and your own records in SageMaker Experiments. The same applies on GCP with Dataplex and on Azure with Purview. The gap is real and is paid for in operations or in broken lineage.

Stage 2 — Tune

The problem. Producing a new model_id, model_version, typically a LoRA adapter over a stable base, as the post on continuous fine-tuning explains, with lineage back to the dataset that trained it and experiment tracking that lets you reproduce it six months later.

OSS stack. Technical core: HuggingFace Transformers + PEFT (LoRA, QLoRA), bitsandbytes for quantisation, DeepSpeed or FSDP for parallelism. Experiment tracking: MLflow (authoritative) or Weights & Biases self-hosted. Convenience frameworks: Axolotl and Llama Factory wrap the machinery above in declarative configuration. Distributed orchestration: Kubeflow Training Operator or Ray Train. On small infrastructures, direct scripts with Slurm or K8s Jobs over GPU pools. The lineage chain dataset → run → model is closed by registering the dataset as an MLflow input artefact.

Hyperscaler equivalents. On AWS, SageMaker Training Jobs covers most workloads, SageMaker HyperPod covers large training runs with resilience to node failures, and SageMaker JumpStart offers click-to-train fine-tuning over a catalogue of pre-curated models. For fine-tuning Bedrock models (hosted Claude, Llama, Mistral) there is Bedrock Custom Models: you upload the dataset to S3, Bedrock trains, and it returns a private endpoint with provisioned throughput. The equivalent experiment tracking is SageMaker Experiments or managed MLflow in SageMaker (yes, AWS has officially hosted MLflow since 2024).

On GCP, Vertex AI Custom Training runs any container with GPUs or TPUs; Vertex AI Tuning is the managed API for fine-tuning Gemini and Model Garden models. Experiment tracking in Vertex AI Experiments (with MLflow compatibility).

On Azure, Azure ML Training Jobs on your own clusters or managed compute; Azure OpenAI fine-tuning for fine-tuning GPT and the o-series; Azure ML Experiments with MLflow natively integrated since 2022.

Summary table — Tune stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Training framework	HF Transformers + PEFT	SageMaker SDK	Vertex AI SDK	Azure ML SDK
Quantisation / parallelism	bitsandbytes, DeepSpeed, FSDP	SageMaker libs + HF support	Vertex + HF support	Azure ML + HF support
Managed fine-tuning (black box)	—	Bedrock Custom Models, JumpStart	Vertex Tuning (Gemini)	Azure OpenAI fine-tuning
Distributed on a cluster	Kubeflow, Ray Train, Slurm	SageMaker HyperPod	Vertex AI Training (multi-node)	Azure ML compute clusters
Experiment tracking	MLflow, W&B self-hosted	SageMaker Experiments, managed MLflow	Vertex Experiments	Azure ML + MLflow
Access to the base model	Whatever you download (Llama, Mistral, Qwen)	Bedrock catalog + HF Hub	Vertex Model Garden + HF Hub	Azure ML model catalog + HF Hub

Where the names mislead. The managed fine-tunings (Bedrock Custom, Vertex Tuning, AOAI fine-tuning) are a black box: you do not choose hyperparameters beyond a handful, you do not see the trainer’s detailed logs, and you cannot inspect the dataset once it is inside their pipeline. The experiment tracking they offer is not comparable to MLflow sitting next to the trainer, where you can capture any metric and artefact. For scenarios where operationally you do not need inspection this is liberating; for ENS / NIS2 scenarios where you have to prove what trained what, the black box fails by construction.

Stage 3 — Eval

The problem. Validating candidates before and after promotion against a golden set, with operational metrics (faithfulness to the RAG, tone, format compliance, toxicity, jailbreak resistance, PII leakage) run as gates in CI and as online sampling. Covered in the post on evals and the one on guardrails.

OSS stack. Eval suites: DeepEval, RAGAS (specialised in RAG), Promptfoo (declarative, ideal for CI), lm-eval-harness (academic), HELM. Evals integrated with tracing: Langfuse Evals, Phoenix Arize OSS. LLM-as-judge judges: any local OSS model; in serious systems, two different judges to reduce bias. Safety and guardrails: NeMo Guardrails (NVIDIA), Guardrails AI, LlamaGuard + PromptGuard (Meta), ShieldGemma (Google, open weights), PII detectors such as Presidio (Microsoft) on-prem.

Hyperscaler equivalents. On AWS, Bedrock Model Evaluation offers automatic evals (toxicity, accuracy, robustness) and human-in-the-loop, Bedrock Guardrails covers the safety layer (denied topics, PII, prompt injection, contextual grounding check), and SageMaker Clarify adds bias and explainability over general models.

On GCP, Vertex AI Evaluation Service runs evals with automatic metrics and an LLM judge, Vertex AI Model Armor and the safety filters built into the Gemini API cover the guardrails layer. Vertex AI Studio exposes interactive Eval for prompt iteration.

On Azure, the Azure AI Evaluation SDK runs offline evals against datasets, Azure AI Content Safety covers safety (Prompt Shields against jailbreak, Groundedness detection, content categories, PII detection). All accessible from Azure AI Foundry.

Summary table — Eval stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Automatic eval suite	DeepEval, RAGAS, Promptfoo	Bedrock Model Evaluation	Vertex AI Evaluation Service	Azure AI Evaluation SDK
LLM-as-judge	Any OSS model	Bedrock judge models	Vertex judge (Gemini)	Azure OpenAI judges
Golden set management	Langfuse datasets, manual	SageMaker Ground Truth datasets	Vertex Datasets	Azure ML Datasets
Guardrails (jailbreak, PII, prompt injection)	NeMo Guardrails, LlamaGuard, Presidio	Bedrock Guardrails	Vertex Model Armor + Gemini safety	Azure AI Content Safety (Prompt Shields, Groundedness)
Eval in CI	Promptfoo + GitHub Actions	Bedrock Eval API + CodeBuild	Vertex Eval API + Cloud Build	Azure AI Eval + Azure Pipelines

Where the names mislead. Managed guardrails are convenient but opaque: the Bedrock Guardrails rules are configurable but the detection implementation cannot be inspected; the same goes for Azure AI Content Safety. In OSS, NeMo Guardrails shows you the Colang graph and Presidio shows you the recognisers, auditable and modifiable. For regulated systems where an auditor asks “how exactly does it detect PII?”, OSS answers with code; the cloud answers with documentation.

Stage 4 — Deploy

The problem. Serving tokens to the end user with predictable latency and throughput, a decent cost / token ratio, hot-swap adapter support, and multi-tenancy if the business demands it. Covered in the posts on KV cache, PagedAttention, disaggregated serving, vLLM on K8s, LLM operators and multi-tenant cluster.

OSS stack. Inference engine: vLLM (PagedAttention, prefix caching, LoRA hot-swap, OpenAI-compatible API) as the reference, TensorRT-LLM for maximum optimisation on Hopper / Ada, SGLang for workloads with heavy prompt restructuring, TGI (Hugging Face) as a mature alternative, llama.cpp for edge and CPUs, NVIDIA Dynamo for multi-node disaggregated serving on large clusters. Orchestration on Kubernetes: KServe, KubeRay, dedicated operators such as llm-d, vLLM Production Stack and KAITO. Gateway / control plane: Envoy AI Gateway, LiteLLM Proxy, Portkey AI Gateway, Kong AI Gateway. Triton Inference Server covers mixed workloads (LLM + traditional) where a single backend matters.

Hyperscaler equivalents. On AWS, two different routes. The managed-by-model route is Amazon Bedrock: a catalogue of hosted models (Claude, Llama, Mistral, Cohere, Titan), pay per token or Provisioned Throughput with an SLA, Bedrock Prompt Caching as the conceptual equivalent of vLLM’s prefix caching, and integrated Bedrock Agents and Bedrock Knowledge Bases. The managed-by-infrastructure route is SageMaker Endpoints (real-time, async, serverless, batch) with Inference Components to densify multiple models on one instance. Own hardware: AWS Inferentia and Trainium via the Neuron chip, an alternative to NVIDIA with a better cost / token on stable workloads if you compile your model.

On GCP, Vertex AI Prediction Endpoints runs your containers or models from the Model Garden, the Gemini API via Vertex AI offers the managed Gemini models, and Cloud TPU v5e / v5p / Trillium (v6) is the in-house hardware competing with the H100 for training and inference. For sovereignty there is Google Distributed Cloud air-gapped, which brings Vertex AI to a certifiable on-premise rack.

On Azure, Azure OpenAI Service serves OpenAI models (GPT-4.1, o-series, GPT-image), Azure ML Managed Online Endpoints runs any model (OSS included, via container), and Azure AI Foundry models absorbed in 2025 the catalogue of open models served as-a-service. Hardware: Azure ND H100 v5, ND H200 v5, ND GB200 v6 and the in-house bet Microsoft Maia 100 for internal inference.

Summary table — Deploy stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Inference engine	vLLM, TensorRT-LLM, SGLang, TGI	Bedrock (managed model), SM Endpoints (your container)	Vertex Prediction, Gemini API	Azure OpenAI, Azure ML Endpoints
Prefix / prompt caching	vLLM native	Bedrock Prompt Caching	Vertex AI context caching	Azure OpenAI prompt caching
Adapter hot-swap (LoRA)	vLLM `--enable-lora`, S-LoRA	Bedrock Custom Models endpoints	Vertex Tuning endpoints	Azure OpenAI fine-tuned deployments
Disaggregated serving	NVIDIA Dynamo, vLLM PD-disagg	— (managed internally, not exposed)	— (managed internally, not exposed)	— (managed internally, not exposed)
Accelerator hardware	NVIDIA H100/H200/B200, AMD MI300	Inferentia, Trainium, NVIDIA	TPU v5/v6, NVIDIA	Maia, NVIDIA
AI Gateway / proxy	Envoy AI Gateway, LiteLLM, Portkey, Kong	API Gateway + Bedrock	Vertex AI + Apigee	Azure API Management + AOAI
K8s orchestration	KServe, KubeRay, llm-d, KAITO	EKS + SageMaker Operators	GKE + Vertex AI	AKS + KAITO

Where the names mislead. Bedrock Prompt Caching and Vertex context caching sound equivalent to vLLM’s prefix caching, but operationally they are different: the cache lives on the hyperscaler’s plane, its eviction policy is opaque, its cost is billed separately, and you cannot easily see the hit ratio per tenant. In vLLM you see the hit ratio in Prometheus metrics and you decide the policy. The same applies to disaggregated serving: the hyperscalers implement it internally to cut their own cost of serving, but they do not expose prefill/decode control to the user. If you need your workload’s TTFT to be controlled separately from TPS, that lever is not available.

Stage 5 — Observe

The problem. End-to-end LLM traces with trace_id propagated through every component, runtime metrics per tenant, online scoring (LLM judge over sampling), statistical drift, and safety / guardrails monitoring. Covered in the posts on AgentSight tracing, MCP observability with OTel and eBPF + drift.

OSS stack. Base standard: OpenTelemetry (specification + collector + SDKs) with the gen_ai semantic conventions that stabilised in 2025. Backends: Tempo or Jaeger for traces, Prometheus for metrics, Loki for logs, Grafana as the common UI. LLM-specific layer: Langfuse (self-hosted with an optional EE licence) and Phoenix Arize OSS. eBPF layer for low-level observability: Pixie, Hubble, and Cilium Tetragon for runtime security. Drift: Evidently AI, NannyML, Alibi Detect.

Hyperscaler equivalents. On AWS, CloudWatch (metrics + logs) + AWS X-Ray (traces) form the base, CloudWatch Application Signals adds OTel-compatible APM, and Amazon Managed Prometheus and Amazon Managed Grafana serve the plane if you want to keep Prom + Grafana without operating them. Bedrock logging integrates with CloudWatch and S3. ADOT (AWS Distro for OpenTelemetry) is the official collector.

On GCP, Cloud Monitoring + Cloud Logging + Cloud Trace + Cloud Profiler form the quintet, all OTel compatible. Vertex AI Model Monitoring offers drift detection (feature skew, prediction drift) integrated with runs.

On Azure, Azure Monitor + Application Insights + Log Analytics cover the APM stack with native OTel, Azure ML Model Monitor adds drift and data quality, and Azure OpenAI diagnostic logs enrich the traces with token and model metadata.

Summary table — Observe stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Traces (OTel)	OTel + Tempo / Jaeger	X-Ray + ADOT, App Signals	Cloud Trace	App Insights + Azure Monitor
Metrics	Prometheus + Grafana	CloudWatch + AMP / AMG	Cloud Monitoring	Azure Monitor Metrics
Logs	Loki, ELK	CloudWatch Logs	Cloud Logging	Log Analytics
LLM-specific (prompt, scores, sessions)	Langfuse, Phoenix Arize OSS	Bedrock logging + CW + custom	Vertex AI tracing + custom	App Insights + AOAI logs + custom
Drift detection	Evidently, NannyML, Alibi Detect	SageMaker Model Monitor	Vertex AI Model Monitoring	Azure ML Model Monitor
eBPF / runtime	Pixie, Hubble, Tetragon	— (no direct equivalent)	GKE Dataplane v2 / Cloud Service Mesh	Azure CNI + Defender for Cloud

Where the names mislead. The classic cloud APM tools (X-Ray, Cloud Trace, App Insights) do not understand prompt versioning or adapter id as native concepts. They accept the gen_ai.* attributes as dimensions, but the UIs do not prioritise those views. Langfuse and Phoenix do, because they are designed for LLM. In cloud, the usual pattern is to send dual: APM to the managed service for infrastructure + self-hosted Langfuse / Phoenix for the LLM plane. That pays off.

Stage 6 — Retrain + cross-cutting pieces

The problem (Retrain). Closing the loop feedback → triage → enriched dataset → new adapter, with a mixed cadence (quarterly + incident-driven). Covered in the Retrain post.

OSS Retrain stack. Orchestration: Apache Airflow, Prefect, Dagster or Argo Workflows and Kubeflow Pipelines for K8s-native. Feature store where applicable: Feast. Annotation and human-in-the-loop: Argilla, Label Studio, Trubrics. Structured feedback capture: your own Postgres table + Langfuse scores + Phoenix annotations. Closed-loop lineage: OpenLineage tying dataset → run → model → deployment → feedback → next dataset.

Hyperscaler Retrain equivalents. On AWS, SageMaker Pipelines orchestrates the cycle, SageMaker Ground Truth and A2I (Augmented AI) manage annotation and HiL, and SageMaker Model Monitor fires alerts that can invoke retrain pipelines. AWS Step Functions serves as a more general alternative orchestrator.

On GCP, Vertex AI Pipelines (based on Kubeflow Pipelines, compatible) orchestrates, Vertex AI Data Labeling Service annotates, Vertex AI Feature Store manages features, and Workflows or Cloud Composer (managed Airflow) act as orchestration alternatives.

On Azure, Azure ML Pipelines orchestrates, Azure ML Data Labeling annotates, Azure ML Feature Store manages features.

The problem (cross-cutting: prompt + data versioning). Making sure prompt_id, prompt_version and dataset_id, dataset_version propagate through the whole system and show up in spans, runs and metrics. Covered in the posts on prompt versioning and data versioning.

Prompt versioning equivalents. OSS: Langfuse Prompts, MLflow Prompt Registry. AWS: Bedrock Prompt Management (catalogue, versions, labels, integrated A/B testing) and SageMaker Prompt Hub. GCP: Vertex AI Prompt Management inside Vertex AI Studio. Azure: Azure AI Foundry Prompt flow and prompt versioning in Azure OpenAI deployments.

Data versioning equivalents. OSS: DVC + lakeFS (already covered under Data). AWS: S3 Versioning + Lake Formation + Glue Catalog (they are not DVC but together they cover part of it). GCP: Cloud Storage versioning + Dataplex (same). Azure: ADLS Gen2 versioning + Purview (same). The real gap here is that no hyperscaler offers DVC natively, so dataset-as-first-class-citizen operations still require a layer of your own.

Summary table — Retrain stage + cross-cutting pieces.

Functional piece	OSS on-premise	AWS	GCP	Azure
ML pipeline orchestration	Airflow, Dagster, Argo, Kubeflow	SageMaker Pipelines, Step Functions	Vertex AI Pipelines, Cloud Composer	Azure ML Pipelines
Feature store	Feast	SageMaker Feature Store	Vertex AI Feature Store	Azure ML Feature Store
Annotation / HiL	Argilla, Label Studio	SageMaker Ground Truth, A2I	Vertex Data Labeling	Azure ML Data Labeling
Feedback capture	Postgres + Langfuse scores	Bedrock + custom + Ground Truth	Vertex + custom	App Insights + custom
Prompt versioning	Langfuse Prompts, MLflow Prompts	Bedrock Prompt Management	Vertex Prompt Management	Azure AI Foundry Prompt flow
Data versioning	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation	GCS + Dataplex	ADLS + Purview
Cross-system lineage	OpenLineage + DataHub	SageMaker Lineage Tracking	Dataplex lineage	Purview

The chatbot from the previous post ported to AWS

To stop the catalogue being abstract, we take the full scenario from the previous post, the multi-tenant support chatbot for insurers on an on-premise OSS stack, and describe it component by component with an AWS stack. It is not an executable migration; it is the map of what disappears, what appears and where the lock-in shows up.

The network plane. Edge LB and WAF: AWS WAF + CloudFront. Ingress to the cluster: AWS Load Balancer Controller over EKS. What was Cilium BGP + RKE2 is replaced by EKS with VPC CNI (or Cilium on EKS, which is possible). The conceptual equivalent of Tetragon is Amazon GuardDuty for EKS + optional Falco. Moderate lock-in: network control couples to VPC.

The chat gateway and auth. What was a bespoke API gateway with JWT verification materialises as Amazon API Gateway + Amazon Cognito (or IAM Identity Center if it is B2B). The gateway’s AI-aware routing is covered with Bedrock + per-customer tags, or with AWS API Gateway custom authorizers invoking a Lambda for tenant resolution. High lock-in on the identity layer if Cognito is chosen.

The inference engine. Three different options, with a clear trade-off.

Bedrock with a managed model (Claude / Llama / Mistral): all the operations of vLLM, K8s Operators, KV cache and disaggregated serving disappear. You move to Provisioned Throughput for a latency guarantee. You gain time-to-market; you lose control over prefill/decode, over custom LoRA adapters (Bedrock accepts Bedrock-managed fine-tunes but not arbitrary LoRAs), and you enter model lock-in (moving from Claude to Llama means changing API).
SageMaker Endpoints with your own vLLM container: you keep vLLM and its optimisations, but K8s disappears and SageMaker replaces it as the orchestration plane. Inference Components allow you to densify multiple adapters. KV cache, prefix caching and LoRA hot-swap work the same. Moderate lock-in on the SageMaker SDK and on the Inference Components format.
EKS with vLLM (the minimalist option): basically the OSS stack but with EKS instead of RKE2 and EBS/EFS instead of Ceph. Low lock-in, limited benefit from the cloud.

Data layer. The corpus moves to S3 with versioning, the embeddings to Amazon OpenSearch Service or to Aurora pgvector. The radical managed option is Bedrock Knowledge Bases: you upload documents to S3, it indexes them, it exposes a retrieval API. We remove Qdrant, we remove manual embedding pipelines, we remove part of Kafka + Flink. But control over custom reranking, fine-grained per-chunk ACL and the possibility of re-embedding with a new encoder of your own disappears. Bedrock KB uses the Titan or Cohere embedders available in Bedrock, and changing them means changing the whole index. ENS compliance: you have to validate that the buckets and the index live in EU regions and that the embedding model does too.

Stream + CDC. Kafka + Debezium is replaced by MSK + MSK Connect or by Kinesis + DMS. Schema Registry: Glue Schema Registry. The events remain functionally equivalent. Moderate lock-in if you go to Kinesis (Kinesis is not Kafka), low if you go to MSK (Kafka compatibility).

Data versioning. Here the gap is clear. S3 Versioning + Lake Formation + Glue Catalog is not DVC. To preserve the discipline of the previous post, (dataset_id, dataset_version, sha256_hash) propagated as an input artefact to the trainer, you can keep DVC over S3 (DVC works perfectly with S3 as a remote) or accept the limitation and register lineage manually in SageMaker Lineage Tracking. The first option preserves the practice; the second accepts degradation.

Tune stage. The LoRA adapter customer_support_v7 is trained with SageMaker Training Jobs on ml.p5.48xlarge instances (8× H100), using a standard HuggingFace + PEFT container. MLflow managed by SageMaker or your own MLflow on EC2 covers tracking. Alternative: if the black box is acceptable, Bedrock Custom Models with a dataset in S3 produces a fine-tuned Bedrock model without instantiating a GPU manually, at the cost of not being able to inspect the run.

Eval stage. Promptfoo + RAGAS in CI runs the same over CodeBuild. Bedrock Model Evaluation replaces a good part of the automatic eval suite. Bedrock Guardrails replaces NeMo Guardrails + Presidio + LlamaGuard, with the loss of transparency noted earlier.

Deploy stage. If Bedrock was chosen as the engine, this stage vanishes: Bedrock serves. If SageMaker Endpoints + vLLM was chosen, KServe is replaced by SageMaker Operators (or KServe is kept over EKS). The AI Gateway that in OSS was Envoy AI Gateway or LiteLLM becomes API Gateway + Bedrock or API Gateway + Lambda + SageMaker.

Observe stage. The OTel Collector remains the standard. Traces to AWS X-Ray + CloudWatch Application Signals. Metrics to Amazon Managed Prometheus. Logs to CloudWatch Logs + optionally OpenSearch for search. Langfuse is hosted on ECS Fargate or EKS because the cloud has no native equivalent of integrated prompt + traces + scores. Drift: SageMaker Model Monitor replaces Evidently / NannyML. eBPF (Pixie / Hubble / Tetragon) has no direct equivalent in managed AWS, so Falco or installing Tetragon on EKS is still the route.

Retrain stage. SageMaker Pipelines orchestrates the quarterly cycle. SageMaker Ground Truth + A2I replace Argilla. The feedback_signals table in Postgres stays as it is (RDS Postgres) or moves to DynamoDB for large scales.

How much the lock-in weighs. The component with the highest lock-in is Bedrock + Bedrock Knowledge Bases + Bedrock Guardrails: leaving that requires rewriting the inference plane and reindexing the whole RAG. Next is the SageMaker SDK (Pipelines, Endpoints, Training), where leaving costs but means rewriting scripts, not data. Data in S3 is portable (S3 → MinIO with rclone works). OTel observability is portable at almost no cost if the collector is kept as the abstraction. The auth gateway is the other high lock-in point if it uses Cognito.

What you gain. A dramatic reduction in GPU infrastructure operations, K8s patching, CUDA driver management, prefill/decode sizing, Ceph / MinIO management. A very short start-up curve: one request served in less than a sprint versus several weeks of bring-up for the OSS stack. Explicit provider SLAs.

What you lose. Contractual data sovereignty (the data stays in EU regions if configured that way, but the operator is a US third party under the Cloud Act). Visibility of the whole stack (Bedrock is a black box from the model downwards). Roadmap independence (the decision to discontinue a model, raise prices or change guardrails is not controlled by the customer). Fine-grained cost-per-token optimisation (the levers are the ones the provider exposes). For ENS customers under a HIGH declaration or NIS2 essential category, several of these points are non-compliance, not preference.

Master table: the entire parallel catalogue

Stage / component	OSS on-premise (the blog’s reference)	AWS	GCP	Azure
Data	DVC + lakeFS + MinIO + Qdrant + Kafka + Debezium	S3 + Lake Formation + OpenSearch / Aurora pgvector + MSK + DMS	GCS + Dataplex + Vertex Vector Search + Pub/Sub + Datastream	ADLS Gen2 + Purview + Azure AI Search + Event Hubs + ADF
Data versioning (cross-cutting)	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation + Glue Catalog	GCS Versioning + Dataplex lineage	ADLS versioning + Purview
Tune	HF Transformers + PEFT + bitsandbytes + MLflow + Ray/Kubeflow	SageMaker Training + HyperPod + Bedrock Custom + SM Experiments	Vertex AI Training + Vertex Tuning + Vertex Experiments	Azure ML Training + Azure OpenAI fine-tuning + Azure ML + MLflow
Eval	DeepEval + RAGAS + Promptfoo + Langfuse Evals + NeMo Guardrails	Bedrock Model Evaluation + Bedrock Guardrails + SageMaker Clarify	Vertex AI Evaluation Service + Model Armor + Gemini safety	Azure AI Evaluation SDK + Content Safety (Prompt Shields, Groundedness)
Deploy	vLLM + KServe + LLM Operators + Envoy AI Gateway	Bedrock + SageMaker Endpoints (+ Inferentia / Trainium)	Vertex AI Prediction + Gemini API (+ TPU)	Azure OpenAI + Azure ML Endpoints (+ Maia)
Observe	OTel + Tempo + Prometheus + Loki + Langfuse + Phoenix + Hubble	CloudWatch + X-Ray + ADOT + AMP/AMG + SM Model Monitor	Cloud Monitoring + Cloud Trace + Vertex Model Monitoring	Azure Monitor + App Insights + Azure ML Model Monitor
Retrain	Airflow / Argo / Kubeflow Pipelines + Argilla + Feast	SageMaker Pipelines + Ground Truth + A2I + SM Feature Store	Vertex AI Pipelines + Data Labeling + Vertex Feature Store	Azure ML Pipelines + Data Labeling + Azure ML Feature Store
Prompt versioning (cross-cutting)	Langfuse Prompts + MLflow Prompt Registry	Bedrock Prompt Management + SM Prompt Hub	Vertex AI Prompt Management	Azure AI Foundry Prompt flow

When to choose each side — the real decision

The right question is not “OSS or cloud?”. It is stage by stage.

The OSS side wins by default when there is:

Data subject to ENS HIGH category, NIS2 essential sectors or equivalents (identifiable health data, regulated banking, critical infrastructure). Here provider traceability and the processing contract are not negotiable; using a service whose operator is subject to the Cloud Act, FISA 702 or equivalent compromises the legal basis.
A requirement for auditable inspection of the model, the guardrails and the whole pipeline. If a regulator asks “how exactly does it detect PII?” and an answer that ends in open code is mandatory.
Large volumes with stable workloads. Above a certain threshold of tokens per month, the cost of Bedrock / AOAI / Vertex diverges from the amortised cost of your own GPU cluster. The threshold depends on the workload, but it typically sits between 5 and 50 billion tokens per month for models in the Llama 70B range.
Roadmap independence as a priority. The day the provider discontinues a model or raises the price by 40 %, the organisation has to be able to ignore it.

The hyperscaler side wins by default when there is:

Critical time-to-market, an MVP in weeks. The operational weight of the OSS stack is too much for a project that has not yet proved product-market fit.
A small team with no SREs / MLEs specialised in GPU inference. Operating KServe + vLLM + KV cache + multi-tenancy is not trivial; if the team cannot sustain it, hosting is the way.
Variable workloads / unpredictable spikes. Bedrock on-demand and SageMaker serverless charge for what you use; your own cluster pays for the GPU whether it is busy or not.
A need for specific proprietary models (Claude, GPT-4.1, Gemini Pro) with no acceptable OSS equivalent for the case.

Mixed stages are frequent and reasonable. In practice, a common pattern in 2026 is: data, observe and retrain in self-hosted OSS (lineage and sovereignty), tune in OSS on your own cluster, eval in OSS + managed guardrails depending on the safety profile, deploy managed for proprietary models and self-hosted for open models. The question to ask for each stage is: “if the provider raises prices by 50 % or discontinues a component tomorrow, how much does it cost to move it?”. The parallel catalogue in this post gives the answer for each box.

What we have not covered (yet)

Some pieces still deserve a post of their own:

OpenAI / Anthropic API directly (not through Bedrock or AOAI): another level of managed, another contract.
Serious hybrids: AWS Outposts, GCP Distributed Cloud, Azure Stack HCI / Azure Local, the hyperscaler in your own room.
Per-tenant cost accounting compared OSS vs cloud: how the bill is built and where attribution breaks.
A real migration OSS → cloud or cloud → OSS: steps, scripts, gotchas.
Concrete European sovereignty: GAIA-X, EuroHPC, the European cloud offering (OVHcloud, Scaleway, IONOS, Aruba), compared with the big three for ENS / NIS2 cases.
AWS Inferentia / Trainium, GCP TPU v6 Trillium, Azure Maia: in-house chips and how they change the cost / token calculation.

References

vLLM documentation and vLLM Production Stack — the reference OSS inference engine.
Amazon Bedrock documentation — AWS managed model catalogue, Knowledge Bases, Guardrails and Prompt Management.
Amazon SageMaker AI — training, endpoints, pipelines, model monitoring.
Google Vertex AI documentation — training, prediction, evaluation, model monitoring.
Azure AI Foundry documentation — Microsoft’s unified plane for AI applications.
Azure OpenAI Service documentation — OpenAI models hosted on Azure.
OpenTelemetry GenAI semantic conventions — the standard that stitches observability across the OSS / cloud boundary.
Langfuse documentation and Arize Phoenix — the reference OSS LLM observability tools.
DVC and lakeFS — OSS data versioning.
NeMo Guardrails — OSS safety + dialog policy.
ENS (Esquema Nacional de Seguridad) and NIS2 (Network and Information Security Directive 2) — the compliance frameworks that have the last word in the OSS vs cloud choice for regulated EU customers.

Llmops-Pipeline on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

TL;DR

You are here: every stage at once

The analogy: forensic analysis of a request

The scenario: a multi-tenant support chatbot for regulated customers

Step 0 — The frame: the answer the user sees

Rewinding backwards: what was already in place before the request

t = −90 days — The previous Retrain stage closes the earlier cycle

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

t = −31 days — The Deploy stage: the v7 adapter enters serving

Moving forward: the user’s request crosses the system

t = 0 ms — Entry through the gateway

t = 8 ms — Pulling the versioned prompt

t = 12 ms — RAG retrieval

t = 40 ms — Building the final payload

t = 45 ms — Prefill

t = 580 ms — Decode (streaming)

t = 4,200 ms — Complete answer, root span closed

In parallel: Observe is watching

The feedback: the loop closes

Insertion into feedback_signals

Triage by root cause

The next cycle picks it up

What goes in each trace: identity and traceability

Synthesis diagram: how the pieces fit

The on-premise stack applied

Ten bridges between stages where the system breaks

How to walk through the blog

What we have not covered (yet)

See also

References

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

TL;DR

You are here: every stage, but by OSS column

The analogy: the electrician’s toolbox

Diagram of the connected reference OSS stack

Stage 1 — Data + cross-cutting Data versioning

DVC (Data Version Control)

lakeFS

MinIO

Qdrant

PostgreSQL + pgvector

Apache Kafka + Debezium

Apache Flink (brief mention)

Stage 2 — Tune

HuggingFace Transformers + PEFT

bitsandbytes

MLflow Tracking

Axolotl

Ray Train

Stage 3 — Eval + Guardrails

DeepEval

RAGAS (RAG Assessment)

Promptfoo

NeMo Guardrails

Microsoft Presidio

Stage 4 — Deploy

vLLM

KServe

Triton Inference Server

Envoy AI Gateway

llama.cpp

Stage 5 — Observe

OpenTelemetry Collector

Tempo (traces) + Jaeger

Prometheus + Grafana

Langfuse

Phoenix Arize OSS

Cilium Tetragon + Hubble

Evidently AI

Stage 6 — Retrain + cross-cutting pieces

Apache Airflow

Argo Workflows

Kubeflow Pipelines

Feast

Argilla

Langfuse Prompts + MLflow Prompt Registry