The six-stage LLMOps pipeline: global architecture and a deep dive into every component

TL;DR

The first two posts in the series established the LLMOps landscape and went down into the detail of the data pipeline with Kafka. This post does the intermediate zoom: it draws the complete map of the system, a global architecture of a modern LLMOps setup with all the pieces the field has settled on in 2026, and goes into depth on each of the six canonical stages of the pipeline: Data, Tune, Eval, Deploy, Observe, Retrain. For each stage we give the operational sub-tasks, the dominant tools, the design decisions that always show up, and the specific traps that are seen repeatedly in production. And, most important operationally: every stage carries a “you are here” mini-map over the cycle, which will be reused in any later post in the series to place the reader. The idea: that anyone reading a post about fine-tuning, about prompt versioning, about eval gates or about drift detection can look at the mini-map and know immediately which piece of the larger system they are thinking about that day.

This is the third post in the MLOps series specific to LLMs. Previous ones: Landscape 2026 and RAG over Kafka. Here we move from “the what” and “one piece” to the whole map, with detail per stage.

The global architecture: the master map

Before going down into each stage, let us fix the whole map. What follows is the reference drawing of a production LLMOps system in 2026, with every component the field has settled on in its place:

What you see: the six large boxes are the stages; the solid arrows are the flow of the pipeline; the dashed arrow that goes from Retrain to Data is the feedback cycle that turns LLMOps into a living process rather than a project that ends. The grey band at the foot holds the cross-cutting components (observability, prompt versioning, MCP, gateway, schema) that run through every stage and connect to each one.

Three quick readings of the map:

• Horizontal, top: the happy path, data → tune → eval. What happens while you prepare the model.
• Horizontal, bottom: the service path, deploy → observe → retrain. What happens once the model is alive.
• Vertical: the connection between the two floors. Eval gateway feeds Deploy; Observe feeds Retrain; Retrain returns to Data.

From here on, every stage will include a navigation mini-map (“you are here”) to place you in the complete cycle. Let us go through each one.

Stage 1 — Data: ingestion, transport, versioning, indexing

Operational sub-tasks

The Data stage is the most underrated and the one that blocks the most projects. Its sub-tasks:

• Ingestion from heterogeneous sources: OLTP databases (Postgres, MySQL), external APIs, file shares, scraping, SaaS systems, application logs, internal messaging.
• Change capture (CDC) in streaming if the data is dynamic. Debezium over Kafka, Flink CDC, modern alternatives such as RisingWave that reads the WAL directly.
• Transformation (cleansing, dedup, normalisation, PII sanitisation).
• Schema management: schema registry, compatible evolution, backward/forward compatibility.
• Versioning of training datasets and golden datasets: DVC + lakeFS (unified in November 2025). Covered in detail in the dedicated data versioning post.
• Indexing for RAG: chunking, embeddings, writing to vector stores. Covered in depth in the Kafka post.
• Materialisation into analytical tables: Tableflow → Iceberg/Delta, for BI consumption and low-latency queries.

Dominant tools

Design decisions

The three decisions that always show up:

Batch vs streaming: the more dynamic the data, the more streaming. For static corpora (manuals that never change) a nightly batch is enough; for transactional data the agent needs to see minute by minute, streaming from day 1.

Embedding model: changing the embedding model invalidates every indexed vector. An architectural decision: pin the model and have an explicit migration plan (the dual-index pattern seen in the Kafka post).

Vector store: pgvector if you already run Postgres and you are below 10M vectors; Qdrant if you want mid-scale simplicity; Milvus if you need billions; Weaviate if you value native hybrid search.

Traps

• Hardcoding connections to the source (with no abstraction): when the database changes (version, host, schema), you break the whole pipeline. An adapter layer from day 1.
• No schema registry: topics start breaking silently.
• Full reindexing whenever something changes: it costs hours or days. Design the dual-index pattern from the start.
• Unsanitised PII: the RAG is serving sensitive data without meaning to. Anonymisation in the pipeline, not at consumption time.

Stage 2 — Tune: preparing the model for your case

Operational sub-tasks

• Base model selection: Llama, Qwen, Mistral, Gemma, DeepSeek according to licence, size, quality in your domain.
• Dataset preparation: train/val/test split, format (chat templates, JSONL), augmentation where it applies.
• Adapter configuration: LoRA rank, target modules, alpha; QLoRA if you want to train on a consumer GPU; full fine-tune only if you have the budget.
• Training loop: HuggingFace Transformers + PEFT + TRL as the canonical stack; Axolotl or llama-factory as convenience wrappers; Unsloth if you want 2-4× more speed on consumer GPUs.
• Hyperparameter sweep: W&B Sweeps, Optuna, Ray Tune.
• Checkpointing and resumability: save every N steps, resume from failure.
• Promotion: the adapter is promoted to the registry after passing the next stage (Eval).

The three modes of Tune

The detail of the table we saw in the landscape:

Supervised fine-tuning (SFT) with LoRA/QLoRA. You collect (prompt, ideal-response) pairs and apply SFT with cross-entropy loss. The simplest option. The rule of thumb: 300-3,000 well-curated examples are usually more useful than 50,000 noisy ones.

DPO (Direct Preference Optimization) and RLAIF. Instead of “ideal-response”, you collect (prompt, good_response, bad_response) triples and train the model to prefer the good one. More stable than classic RLHF, same objective. It is what most teams use when they go beyond SFT.

Agent training (RFT / Reinforcement Fine-Tuning, pure RLHF). For cases where the model needs to learn multistep trajectories: when to choose tool A over B, when to ask for confirmation, how to decompose a large task. Far more expensive and complex. OpenAI’s work with RFT set the pattern in 2024-2025; in 2026 it is coming out of the experimental phase.

RAG as an alternative to Tune: although conceptually it is another stage (it lives in Data + Deploy), functionally it competes with fine-tuning for many cases. The 2026 verdict: hybrid is the default (60% of deployments), fine-tune for behaviour plus RAG for volatile knowledge.

Tools

Traps

• Catastrophic forgetting: very aggressive SFT destroys the model’s general capabilities. Keep a small % of the original dataset or use regularisation.
• Overfitting to the golden dataset: the model learns to memorise the eval set. Keep a holdout test set that nobody on the team looks at until the final release.
• Train/serve skew: prompts in training with a different format from production. The same chat template in both.
• LoRA rank too high: it looks like it improves metrics but it inflates the adapter with no real benefit. Start with r=8 or r=16; raise it only if there is evidence.

Stage 3 — Eval: validating before promoting

Operational sub-tasks

Covered in depth in Evals: the layer after tracing. A structured summary for the pipeline:

• Golden dataset curation: 100-500 examples as a minimum, actively maintained with incident cases.
• Evaluators: heuristic (regex, length), semantic (embeddings), LLM-as-judge (G-Eval), human (golden labels).
• Running in CI: block the merge if critical metrics drop by more than X%.
• Running on the platform: over sampled production traffic, persist results, detect long-term regression.
• Judge calibration: 85-90% agreement with humans before accepting the judge as productive.
• Eval gates: explicit thresholds per metric (faithfulness > 0.85, relevancy > 0.80, and so on).

Tools

• CI gates: DeepEval (Apache 2.0, pytest-style), Promptfoo (MIT, CLI), Ragas (RAG-specific), Inspect AI (safety/capability).
• Platform: Langfuse (MIT, complete suite), LangSmith (LangChain), Phoenix (ELv2, OTel), Braintrust.
• Judges: GPT-4 (expensive but the reference), Claude 3.5 Sonnet, Prometheus (OSS, 0.897 correlation), JudgeLM.

Traps

• Aged golden dataset: if it is not updated, it stops reflecting production.
• Contaminated judge: the judge knows the dataset (it appeared in its training).
• Insufficient sample size: fewer than 50 examples makes differences look like noise.
• Runaway costs: G-Eval with GPT-4 over many cases costs thousands of USD a month.
• Forgetting the segment: an average of 0.85 can hide a 0.55 in German.

Stage 4 — Deploy: putting the model into production

Operational sub-tasks

Covered in depth in vLLM on Kubernetes and LLM operators on K8s. A summary for the pipeline:

• Runtime selection: vLLM (default), SGLang (agents with high prefix caching), TensorRT-LLM (pure latency), llama.cpp (edge).
• Operator selection: vLLM Production Stack, KServe, OME (LMSYS), NVIDIA Dynamo, llm-d (CNCF).
• Serving configuration: --tensor-parallel-size, --kv-cache-dtype=fp8, --enable-prefix-caching, --enable-chunked-prefill, --gpu-memory-utilization=0.92.
• Routing between models: LiteLLM as the multi-provider abstraction.
• Release strategy: canary (1% → 10% → 100%), blue-green (all or nothing with fast rollback), shadow (eval in parallel without affecting users).
• Autoscaling with LLM metrics: KEDA + Prometheus over vllm:num_requests_waiting or equivalent.
• Gateway / Inference Extension: Gateway API Inference Extension once it is GA.

Dominant tools

• Serving engines: vLLM, SGLang, TensorRT-LLM, llama.cpp, MLX.
• Operators: OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, KServe.
• Routing: LiteLLM (100+ providers), OpenRouter (managed), LangChain Router.
• GPU primitives: NVIDIA GPU Operator, LeaderWorkerSet (LWS) for multi-pod tensor parallel, KEDA for autoscaling.

Traps

• Naïve rolling update that cuts sessions: maxUnavailable: 0, maxSurge: 1 and terminationGracePeriodSeconds: 120+.
• A short readiness probe that kills pods while they load: startupProbe with failureThreshold: 60.
• HPA on CPU% with no LLM metrics: vLLM batches internally, one replica serves dozens. KEDA on queue depth.
• Unquantised KV cache: --kv-cache-dtype=fp8 is almost always worth it.
• Tensor parallel on GPUs without NVLink: all-reduce saturates PCIe and throughput collapses.

Stage 5 — Observe: seeing what happens in production

Operational sub-tasks

This is the stage we have covered most deeply in previous series: the whole eBPF series (4 posts) and the whole post-tracing series (4 posts) deal with sub-tasks of Observe. A structured summary:

• Tracing: OpenLLMetry/Traceloop, Langfuse, Phoenix, LangSmith. Spans with OTel GenAI semantic conventions (gen_ai.*, mcp.*).
• Metrics: Prometheus + Grafana. TTFT, TPOT, throughput, queue depth, KV cache usage, cost per tool.
• Active guardrails (not just eval): NeMo Guardrails with 5 types of rails, multimodal Llama Guard 4, Llama Prompt Guard 2 (86M/22M), LLM Guard.
• eBPF observability (zero-instrumentation): Hubble (network), Tetragon (process/syscall), AgentSight (LLM agent with SSL uprobes + stdiocap MCP).
• eBPF in the local engine (inference): ProfInfer-style with uprobes in llama.cpp / vLLM / libcudart.
• Drift detection: Evidently AI, NannyML, WhyLabs. KS, PSI, MMD over embeddings.
• MCP observability: OpenTelemetry GenAI MCP semantic conventions, trace propagation via params._meta, a centralised MCP Gateway.

The four mandatory metrics

Out of everything covered, the four any minimal dashboard must have:

1. TTFT p50/p95 (time to first token) — what the user perceives.
2. TPOT p50/p95 (time per output token) — streaming speed.
3. Throughput (aggregate tokens per second) — capacity planning.
4. Queue depth (vllm:num_requests_waiting) — the leading indicator.

To these you add, per domain:

• For RAG: faithfulness rolling mean, retrieval hit rate.
• For agents: tool call accuracy, multi-step task completion.
• For multi-tenant: cost per tenant, p95 latency per tenant.

Traps

• Cardinality in Prometheus: metrics carrying every K8s label blow up.
• Tracing with no sampling: storage grows out of control.
• Guardrails permanently in monitoring mode: they never reach enforce.
• Drift with no alerts: you spot drift on the dashboard once a month; meanwhile the problem has been running for weeks.
• OTel with no propagation: MCP, Tetragon and AgentSight spans left disconnected.

Stage 6 — Retrain: closing the loop

Operational sub-tasks

This is the stage most neglected in GenAI projects. Closing the loop turns LLMOps into a living practice; not closing it leaves it as a project that ages.

• Explicit feedback: thumbs up/down in the UI, annotations by power users, forms for “what went wrong”.
• Implicit feedback: anomalous latency, abandonment rate, user retries, aborted sessions.
• Incident triaging: classify incidents by root cause (model issue, retrieval issue, prompt issue, infra issue).
• Dataset enrichment: add to the golden dataset the cases where the system failed, with the correct answer labelled by a human.
• Retrain cadence: quarterly by default, incident-driven when a problematic pattern goes over a threshold.
• Promotion: the new model/adapter goes through the Tune → Eval → Deploy stages, with eval gates that compare against the model in production.

The two cadences

Scheduled retrain (quarterly or half-yearly): an established process. It lets you plan capacity, budget and risk. The default.

Incident-driven retrain: when a serious incident (detected drift, a failing segment, a prompt injection attack) crosses a threshold, a mini-cycle fires. More expensive but necessary for critical cases.

Dominant tools

• Annotation and feedback collection: Langfuse (built-in UI), Argilla (OSS), Label Studio.
• Dataset enrichment: pipelines in Airflow or Argo Workflows.
• Triaging: Langfuse dashboards with filters on traces with a low eval.
• Promoting candidates: MLflow model registry stages.

Traps

• An open loop: production does not inform the dataset; the model never improves.
• Human feedback gets lost: thumbs down with no structured capture channel.
• Undefined cadence: “we retrain when it is needed” → nobody ever retrains.
• No holdout test set: the golden dataset is enriched with the same cases used to evaluate; eval measures memorisation.
• Promotion with no gates: the new model enters production without passing the checks the previous models passed.

The complete cycle: how the stages fit together

Now that we have seen each stage separately, the key insight is how they hook into each other. Five emergent properties of the cycle:

1. Data is the raw material of every stage. Tune reads from the golden dataset. Eval reads from the eval dataset. Deploy reads from the RAG (vector store). Observe produces new data. Retrain creates new datasets. The Kafka log is the gospel of the whole system (post 2 in the series).

2. Eval is the bidirectional gatekeeper. Before Deploy: it blocks the release if the model regresses. After Observe: it feeds Retrain by identifying the worst-scoring cases. The quality of the eval determines the quality of the whole cycle.

3. Observe feeds Retrain and Eval simultaneously. Traces produce metrics for Observe; problematic traces are annotated and go to the dataset; the new cases enrich the eval golden set. Observe is the operational source of truth.

4. The cross-cutting components (the grey band on the map) are not a stage, they are an infrastructure. OpenTelemetry, prompt versioning, MCP gateway, model gateway, schema registry. Badly configured, every stage suffers separately. Well configured, the stages integrate without friction.

5. The cycle is not strictly sequential, it is concurrent. At any given moment the system has: requests being served (Deploy + Observe), a new version in training (Tune), continuous eval in CI (Eval), data arriving from CDC (Data), incident analysis (Retrain). Every stage is alive at once.

Cross-stage traps: things that break the whole system

There are errors that do not belong to a stage but to the interfaces between stages. The most common ones:

Train/serve skew

The exact format of the prompt in training is different from the one in production. The result: the model trained to answer <|im_start|>user\n...\n<|im_end|> receives User: ...\nAssistant: in production and performs worse. Solution: extract the chat template into a shared library used by the Tune pipeline and the Deploy one.

An eval that does not reflect production

Your golden dataset is careful questions; production is real questions with typos, mixed languages, and so on. Eval passes at 95%, production performs at 70%. Solution: continuously enrich the golden set with real samples.

Drift with no response pipeline

You spot drift on the Observe dashboard; nobody has a defined workflow for what to do. Solution: every drift alert must have a clear runbook: investigate, classify, act (retrain, adjust the prompt, widen the retrieval).

Schema break cascade

You change the schema at the OLTP source; Debezium reflects it; the Flink job breaks; the embedded topic stops updating; the vector store ages; the RAG answers over old data. Three stages affected by one change in Data. Solution: mandatory backward-compatible schema evolution, contracts between producers and consumers.

No observability of the pipeline itself

The LLMOps pipeline is a complex system. If it has no observability of its own (how long training takes, how many jobs fail, how many re-embeddings run), debugging failures is an exercise in spelunking. Solution: OTel over the pipeline itself, not just over the LLM calls.

Invisible vendor lock-in

Pipelines written against LangChain, prompts stuck in LangSmith, embeddings in Pinecone, the model in OpenAI. Migrating is a months-long project. Solution: LiteLLM and OpenLLMetry abstractions, vendor-neutral from the start.

What comes in the following posts

• Post 4 — PostgreSQL + Qdrant in the ingestion stage — the first post that applies the “you are here” pattern over the Data stage. Synchronisation patterns (outbox + CDC), microservices architecture, deployment manifests.
• Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / runtime alignment, continuous fine-tuning in depth, edge LLMs.
• In any later post in this or future series, the “you are here” mini-map will tell you which stage of the cycle the topic fits into. If you read a post about quantization, you will know you are in Deploy. If you read one about evaluator ensembles, you will know you are in Eval. If you read one about RAG over Iceberg, you will know you are in Data.
• If you want to see the whole pipeline in action following a single real request, the synthesis post Anatomy of an LLM request in production does exactly that: it rewinds a request back to the data that trained it 90 days earlier and follows it forward to the feedback that will reappear in the next Retrain cycle, crossing the six stages and the two cross-cutting components in one coherent story.
• If what interests you is comparing how each stage is built in open source against the hyperscalers, the post The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure makes the vertical cut: for every stage, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, gap identification and the multi-tenant chatbot ported to an AWS stack as a concrete example.
• If you want the OSS toolbox piece by piece, the post The OSS catalogue for LLMOps in six stages: entry by entry does the zoom in: ~150 words of description per core tool (what it does, how it differs from its alternatives, licence and governance, typical gotcha), a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.
• If what you are missing is the common vocabulary that runs through the six stages, the post Ontologies and knowledge graphs in LLMOps walks the six stages from the perspective of shared formal nomenclature: how TBox + ABox + SHACL + SKOS change the operation of Data, Train/Adapt, Eval, Deploy, Observe and Govern, the GraphRAG 2026 landscape (Microsoft GraphRAG v2, LightRAG, HippoRAG 2, KAG/OpenSPG), vertical ontologies actually deployed (FIBO, SNOMED CT, schema.org, ENS, EU AI Act) and an open source stack with its licensing caveats (Neo4j Community GPLv3, KuzuDB upstream archived in October 2025).

References

Foundations:

• The Complete MLOps/LLMOps Roadmap for 2026 (Sanjeeb Panda).
• MLOps in 2026: Architecture, Trends & Strategy (Hyscaler).

Per stage (entries from the blog series):

• Data: RAG over Kafka — technical architecture and the dedicated post on data versioning with DVC and lakeFS — the four artefacts to version separately (training, RAG corpus, golden eval, enriched retrain), schema contracts, end-to-end lineage from dataset to trace, and why a golden set without a strict holdout measures memorisation.
• Tune: partly covered in Landscape 2026; a deeper treatment in post 4 if continuous fine-tuning is chosen.
• Eval: Evals: the layer after tracing.
• Deploy: vLLM on Kubernetes and LLM operators on K8s.
• Observe: the whole eBPF series and the whole post-tracing series.
• Retrain: the dedicated post on how to close the loop — feedback capture (explicit + implicit), triage by root cause, dataset enrichment with human annotation (Argilla / Label Studio), scheduled vs incident-driven cadences, governed promotion with eval gates.

Cross-cutting components:

• Prompt versioning: the dedicated post with Langfuse and MLflow Prompts — the three-primitive pattern (immutable version, mutable label, cache), eval gates on promotion, and per-request traceability.
• MCP: deep MCP observability.
• Drift detection: eBPF + drift detection.
• Local inference: PagedAttention deep dive, KV cache.

Frameworks and tools referenced:

• MLflow, W&B, Kubeflow, ZenML, BentoML, Metaflow.
• HuggingFace PEFT, TRL, Axolotl, Unsloth.
• DVC + lakeFS.
• Langfuse, Evidently AI, Phoenix.
• vLLM Production Stack, KServe, OME, NVIDIA Dynamo, llm-d.