The parallel catalogue: the six LLMOps stages in open source and on the hyperscalers (AWS, GCP, Azure)

TL;DR

The previous forensic post used a single request to walk the six stages of the LLMOps pipeline and the two cross-cutting components. This post walks the same stages but crosses them with three extra columns: how each stage is built in open source on-premise, and what the equivalent services are on AWS, GCP and Azure. It is not a migration guide nor a cost benchmark: it is a catalogue of equivalences with its gaps. The general pattern you will see: OSS gives you control, sovereignty and free composition in exchange for expensive operations; the hyperscalers give you integration and time-to-market in exchange for lock-in on margins, data contracts and political dependency. For scenarios subject to ENS / NIS2 with critical customer data, OSS wins by default; for discovery projects where time-to-market is the deciding metric, the hyperscaler wins by default. The interesting part is in the middle. As a concrete thread, at the end we take the multi-tenant chatbot from the previous post and port it to AWS piece by piece to show what disappears, what appears, and where the lock-in materialises.

You are here: the same six stages, but by column

This post shares a map with the previous one, the six stages and the two cross-cutting pieces are all active, but it changes the cut: instead of following a request horizontally, it makes the vertical cut and shows which tools live in each stage depending on the deployment model.

The analogy: your own bakery and the franchise

A baker opens a business. There are two possible models.

He can open his own bakery: rent the premises, buy the oven, choose the flour suppliers, hire his master baker, write his recipes, set the prices, dress the shop window. The day he wants to launch an organic rye sourdough loaf, he asks nobody for permission. The day the price of flour goes up, he finds another supplier. But he pays for all of it: the initial investment, the risk, the daily operations, the months when he misreads the neighbourhood. The bakery is his.

Or he can join a franchise: the franchisor hands over the turnkey premises, the oven with a maintenance contract, the suppliers already negotiated, the operating manuals, the written recipes, the centralised marketing, the loyalty app, the POS system. The learning curve is weeks, not years. But the recipes belong to the franchisor, so do the suppliers, the price of bread is in the catalogue, and the day the croissant formula changes he gets an informational email, not a business decision.

Both bakeries turn out bread. Both meet health regulations and generate income. The operational difference is enormous and it is not about technology: it is about ownership, control and timescale.

The parallel with LLMOps is direct. The on-premise OSS stack is the bakery you own. The managed stack on the hyperscalers is the franchise. The pieces that appear in each stage are functionally equivalent, at the end of the day both solve the same technical problem, but the governance model, the operational cost, the lock-in and the compliance guarantees are different. This post builds the parallel catalogue so that the choice is not made by default.

Quick recap of the previous post

In the forensic post we followed one specific request: a premium-es user of an insurance company asking “How do I cancel my premium subscription?” to the multi-tenant support chatbot of the SaaS provider that hosts it. The route crossed the six stages of the LLMOps pipeline, Data, Tune, Eval, Deploy, Observe, Retrain, plus the two cross-cutting components, prompt versioning and data versioning, over an on-premise infrastructure: RKE2 with Cilium BGP, a 4×H100 SXM cluster, an RTX 4090 for development, vLLM on Kubernetes, Langfuse + OTel + Prometheus + Tempo, Postgres + Qdrant, DVC + lakeFS + MinIO, Kafka and MLflow. The system meets ENS / NIS2 and keeps trace_id propagated end to end.

What comes now is that same system, piece by piece, showing for each box which tool does the job if you are on public cloud, because the integrator’s question is rarely “OSS yes or no?”: it is “what do I lose and what do I gain if I take this box managed?”. And the answer differs box by box.

Stage 1 — Data

The problem. There are three sub-problems the Data stage solves, frequently confused with one another. First, versioning and identity of the corpus and of the training datasets (making sure a dataset_id, dataset_version exists and propagates). Second, storage and serving of the operational corpus (object store + vector index + structured text). Third, streams and ingestion from source systems with CDC, transformation and stable schemas (Schema Registry).

Reference OSS stack. Versioning lives in DVC (pointers in git, content in an object store) combined with lakeFS for branch/merge semantics over data. The post on data versioning goes into the functional difference. The object store is MinIO or Ceph. The vector index is Qdrant or Milvus for large corpora (millions of chunks) and pgvector on Postgres 18 for small cases where having one component fewer pays off operationally. The stream layer is Kafka (pure Apache or Redpanda) with a Schema Registry (Confluent or Karapace OSS), CDC with Debezium or Flink CDC, transformation with Flink or Spark Structured Streaming. The catalogue / lineage is DataHub, Apache Atlas or OpenMetadata with OpenLineage events between systems. The post on PostgreSQL + Qdrant ingestion and the post on RAG over Kafka cover the detailed operations.

Hyperscaler equivalents. On AWS, the corpus lives in S3 (with versioning enabled, which is the cheap substitute for serious data versioning), tabular queries in Athena or Redshift, the vector index in Amazon OpenSearch with the vector plug-in or in Amazon Aurora pgvector. The stream layer is MSK (managed Kafka) or Kinesis Data Streams, CDC with AWS DMS, transformation with Glue Streaming or MSK Connect. The catalogue is AWS Glue Data Catalog + AWS Lake Formation for data governance. And for the RAG case there is also Amazon Bedrock Knowledge Bases, the managed shortcut: you give it S3, it indexes into OpenSearch or Aurora pgvector, exposes a retrieval API and the operations end there, in exchange for paying per indexed chunk and per query.

On GCP, the corpus lives in Cloud Storage (with object versioning), the analytical warehouse is BigQuery (with BigQuery Vector Search already integrated), the dedicated vector store is Vertex AI Vector Search (formerly Matching Engine). The stream layer is Pub/Sub + Dataflow, CDC with Datastream. Catalogue and lineage are Dataplex (which in 2024-2025 absorbed Data Catalog and added automatic lineage). The managed equivalent of Knowledge Bases is Vertex AI Search (formerly Discovery Engine).

On Azure, the corpus lives in ADLS Gen2, tabular queries in Microsoft Fabric / Azure Synapse, the vector index in Azure AI Search (vector mode) or Azure Cosmos DB for PostgreSQL with pgvector. The stream layer is Event Hubs + Stream Analytics or Microsoft Fabric Real-Time Intelligence, CDC with Azure Data Factory. The catalogue is Microsoft Purview, which covers catalogue, lineage and data governance integrated with Entra ID.

Summary table — Data stage.

Where the names mislead. S3 Versioning is not DVC. It keeps object versions but has no notion of a dataset (which objects together make up version 3 of the enriched set?), does not propagate dataset_hash to the trainer, does not integrate with experiment tracking, and does not fail a CI run if a dataset breaks schema. Covering that properly on AWS requires combining it with Lake Formation, Glue Data Catalog and your own records in SageMaker Experiments. The same applies on GCP with Dataplex and on Azure with Purview. The gap is real and is paid for in operations or in broken lineage.

Stage 2 — Tune

The problem. Producing a new model_id, model_version, typically a LoRA adapter over a stable base, as the post on continuous fine-tuning explains, with lineage back to the dataset that trained it and experiment tracking that lets you reproduce it six months later.

OSS stack. Technical core: HuggingFace Transformers + PEFT (LoRA, QLoRA), bitsandbytes for quantisation, DeepSpeed or FSDP for parallelism. Experiment tracking: MLflow (authoritative) or Weights & Biases self-hosted. Convenience frameworks: Axolotl and Llama Factory wrap the machinery above in declarative configuration. Distributed orchestration: Kubeflow Training Operator or Ray Train. On small infrastructures, direct scripts with Slurm or K8s Jobs over GPU pools. The lineage chain dataset → run → model is closed by registering the dataset as an MLflow input artefact.

Hyperscaler equivalents. On AWS, SageMaker Training Jobs covers most workloads, SageMaker HyperPod covers large training runs with resilience to node failures, and SageMaker JumpStart offers click-to-train fine-tuning over a catalogue of pre-curated models. For fine-tuning Bedrock models (hosted Claude, Llama, Mistral) there is Bedrock Custom Models: you upload the dataset to S3, Bedrock trains, and it returns a private endpoint with provisioned throughput. The equivalent experiment tracking is SageMaker Experiments or managed MLflow in SageMaker (yes, AWS has officially hosted MLflow since 2024).

On GCP, Vertex AI Custom Training runs any container with GPUs or TPUs; Vertex AI Tuning is the managed API for fine-tuning Gemini and Model Garden models. Experiment tracking in Vertex AI Experiments (with MLflow compatibility).

On Azure, Azure ML Training Jobs on your own clusters or managed compute; Azure OpenAI fine-tuning for fine-tuning GPT and the o-series; Azure ML Experiments with MLflow natively integrated since 2022.

Summary table — Tune stage.

Where the names mislead. The managed fine-tunings (Bedrock Custom, Vertex Tuning, AOAI fine-tuning) are a black box: you do not choose hyperparameters beyond a handful, you do not see the trainer’s detailed logs, and you cannot inspect the dataset once it is inside their pipeline. The experiment tracking they offer is not comparable to MLflow sitting next to the trainer, where you can capture any metric and artefact. For scenarios where operationally you do not need inspection this is liberating; for ENS / NIS2 scenarios where you have to prove what trained what, the black box fails by construction.

Stage 3 — Eval

The problem. Validating candidates before and after promotion against a golden set, with operational metrics (faithfulness to the RAG, tone, format compliance, toxicity, jailbreak resistance, PII leakage) run as gates in CI and as online sampling. Covered in the post on evals and the one on guardrails.

OSS stack. Eval suites: DeepEval, RAGAS (specialised in RAG), Promptfoo (declarative, ideal for CI), lm-eval-harness (academic), HELM. Evals integrated with tracing: Langfuse Evals, Phoenix Arize OSS. LLM-as-judge judges: any local OSS model; in serious systems, two different judges to reduce bias. Safety and guardrails: NeMo Guardrails (NVIDIA), Guardrails AI, LlamaGuard + PromptGuard (Meta), ShieldGemma (Google, open weights), PII detectors such as Presidio (Microsoft) on-prem.

Hyperscaler equivalents. On AWS, Bedrock Model Evaluation offers automatic evals (toxicity, accuracy, robustness) and human-in-the-loop, Bedrock Guardrails covers the safety layer (denied topics, PII, prompt injection, contextual grounding check), and SageMaker Clarify adds bias and explainability over general models.

On GCP, Vertex AI Evaluation Service runs evals with automatic metrics and an LLM judge, Vertex AI Model Armor and the safety filters built into the Gemini API cover the guardrails layer. Vertex AI Studio exposes interactive Eval for prompt iteration.

On Azure, the Azure AI Evaluation SDK runs offline evals against datasets, Azure AI Content Safety covers safety (Prompt Shields against jailbreak, Groundedness detection, content categories, PII detection). All accessible from Azure AI Foundry.

Summary table — Eval stage.

Where the names mislead. Managed guardrails are convenient but opaque: the Bedrock Guardrails rules are configurable but the detection implementation cannot be inspected; the same goes for Azure AI Content Safety. In OSS, NeMo Guardrails shows you the Colang graph and Presidio shows you the recognisers, auditable and modifiable. For regulated systems where an auditor asks “how exactly does it detect PII?”, OSS answers with code; the cloud answers with documentation.

Stage 4 — Deploy

The problem. Serving tokens to the end user with predictable latency and throughput, a decent cost / token ratio, hot-swap adapter support, and multi-tenancy if the business demands it. Covered in the posts on KV cache, PagedAttention, disaggregated serving, vLLM on K8s, LLM operators and multi-tenant cluster.

OSS stack. Inference engine: vLLM (PagedAttention, prefix caching, LoRA hot-swap, OpenAI-compatible API) as the reference, TensorRT-LLM for maximum optimisation on Hopper / Ada, SGLang for workloads with heavy prompt restructuring, TGI (Hugging Face) as a mature alternative, llama.cpp for edge and CPUs, NVIDIA Dynamo for multi-node disaggregated serving on large clusters. Orchestration on Kubernetes: KServe, KubeRay, dedicated operators such as llm-d, vLLM Production Stack and KAITO. Gateway / control plane: Envoy AI Gateway, LiteLLM Proxy, Portkey AI Gateway, Kong AI Gateway. Triton Inference Server covers mixed workloads (LLM + traditional) where a single backend matters.

Hyperscaler equivalents. On AWS, two different routes. The managed-by-model route is Amazon Bedrock: a catalogue of hosted models (Claude, Llama, Mistral, Cohere, Titan), pay per token or Provisioned Throughput with an SLA, Bedrock Prompt Caching as the conceptual equivalent of vLLM’s prefix caching, and integrated Bedrock Agents and Bedrock Knowledge Bases. The managed-by-infrastructure route is SageMaker Endpoints (real-time, async, serverless, batch) with Inference Components to densify multiple models on one instance. Own hardware: AWS Inferentia and Trainium via the Neuron chip, an alternative to NVIDIA with a better cost / token on stable workloads if you compile your model.

On GCP, Vertex AI Prediction Endpoints runs your containers or models from the Model Garden, the Gemini API via Vertex AI offers the managed Gemini models, and Cloud TPU v5e / v5p / Trillium (v6) is the in-house hardware competing with the H100 for training and inference. For sovereignty there is Google Distributed Cloud air-gapped, which brings Vertex AI to a certifiable on-premise rack.

On Azure, Azure OpenAI Service serves OpenAI models (GPT-4.1, o-series, GPT-image), Azure ML Managed Online Endpoints runs any model (OSS included, via container), and Azure AI Foundry models absorbed in 2025 the catalogue of open models served as-a-service. Hardware: Azure ND H100 v5, ND H200 v5, ND GB200 v6 and the in-house bet Microsoft Maia 100 for internal inference.

Summary table — Deploy stage.

Where the names mislead. Bedrock Prompt Caching and Vertex context caching sound equivalent to vLLM’s prefix caching, but operationally they are different: the cache lives on the hyperscaler’s plane, its eviction policy is opaque, its cost is billed separately, and you cannot easily see the hit ratio per tenant. In vLLM you see the hit ratio in Prometheus metrics and you decide the policy. The same applies to disaggregated serving: the hyperscalers implement it internally to cut their own cost of serving, but they do not expose prefill/decode control to the user. If you need your workload’s TTFT to be controlled separately from TPS, that lever is not available.

Stage 5 — Observe

The problem. End-to-end LLM traces with trace_id propagated through every component, runtime metrics per tenant, online scoring (LLM judge over sampling), statistical drift, and safety / guardrails monitoring. Covered in the posts on AgentSight tracing, MCP observability with OTel and eBPF + drift.

OSS stack. Base standard: OpenTelemetry (specification + collector + SDKs) with the gen_ai semantic conventions that stabilised in 2025. Backends: Tempo or Jaeger for traces, Prometheus for metrics, Loki for logs, Grafana as the common UI. LLM-specific layer: Langfuse (self-hosted with an optional EE licence) and Phoenix Arize OSS. eBPF layer for low-level observability: Pixie, Hubble, and Cilium Tetragon for runtime security. Drift: Evidently AI, NannyML, Alibi Detect.

Hyperscaler equivalents. On AWS, CloudWatch (metrics + logs) + AWS X-Ray (traces) form the base, CloudWatch Application Signals adds OTel-compatible APM, and Amazon Managed Prometheus and Amazon Managed Grafana serve the plane if you want to keep Prom + Grafana without operating them. Bedrock logging integrates with CloudWatch and S3. ADOT (AWS Distro for OpenTelemetry) is the official collector.

On GCP, Cloud Monitoring + Cloud Logging + Cloud Trace + Cloud Profiler form the quintet, all OTel compatible. Vertex AI Model Monitoring offers drift detection (feature skew, prediction drift) integrated with runs.

On Azure, Azure Monitor + Application Insights + Log Analytics cover the APM stack with native OTel, Azure ML Model Monitor adds drift and data quality, and Azure OpenAI diagnostic logs enrich the traces with token and model metadata.

Summary table — Observe stage.

Where the names mislead. The classic cloud APM tools (X-Ray, Cloud Trace, App Insights) do not understand prompt versioning or adapter id as native concepts. They accept the gen_ai.* attributes as dimensions, but the UIs do not prioritise those views. Langfuse and Phoenix do, because they are designed for LLM. In cloud, the usual pattern is to send dual: APM to the managed service for infrastructure + self-hosted Langfuse / Phoenix for the LLM plane. That pays off.

Stage 6 — Retrain + cross-cutting pieces

The problem (Retrain). Closing the loop feedback → triage → enriched dataset → new adapter, with a mixed cadence (quarterly + incident-driven). Covered in the Retrain post.

OSS Retrain stack. Orchestration: Apache Airflow, Prefect, Dagster or Argo Workflows and Kubeflow Pipelines for K8s-native. Feature store where applicable: Feast. Annotation and human-in-the-loop: Argilla, Label Studio, Trubrics. Structured feedback capture: your own Postgres table + Langfuse scores + Phoenix annotations. Closed-loop lineage: OpenLineage tying dataset → run → model → deployment → feedback → next dataset.

Hyperscaler Retrain equivalents. On AWS, SageMaker Pipelines orchestrates the cycle, SageMaker Ground Truth and A2I (Augmented AI) manage annotation and HiL, and SageMaker Model Monitor fires alerts that can invoke retrain pipelines. AWS Step Functions serves as a more general alternative orchestrator.

On GCP, Vertex AI Pipelines (based on Kubeflow Pipelines, compatible) orchestrates, Vertex AI Data Labeling Service annotates, Vertex AI Feature Store manages features, and Workflows or Cloud Composer (managed Airflow) act as orchestration alternatives.

On Azure, Azure ML Pipelines orchestrates, Azure ML Data Labeling annotates, Azure ML Feature Store manages features.

The problem (cross-cutting: prompt + data versioning). Making sure prompt_id, prompt_version and dataset_id, dataset_version propagate through the whole system and show up in spans, runs and metrics. Covered in the posts on prompt versioning and data versioning.

Prompt versioning equivalents. OSS: Langfuse Prompts, MLflow Prompt Registry. AWS: Bedrock Prompt Management (catalogue, versions, labels, integrated A/B testing) and SageMaker Prompt Hub. GCP: Vertex AI Prompt Management inside Vertex AI Studio. Azure: Azure AI Foundry Prompt flow and prompt versioning in Azure OpenAI deployments.

Data versioning equivalents. OSS: DVC + lakeFS (already covered under Data). AWS: S3 Versioning + Lake Formation + Glue Catalog (they are not DVC but together they cover part of it). GCP: Cloud Storage versioning + Dataplex (same). Azure: ADLS Gen2 versioning + Purview (same). The real gap here is that no hyperscaler offers DVC natively, so dataset-as-first-class-citizen operations still require a layer of your own.

Summary table — Retrain stage + cross-cutting pieces.

The chatbot from the previous post ported to AWS

To stop the catalogue being abstract, we take the full scenario from the previous post, the multi-tenant support chatbot for insurers on an on-premise OSS stack, and describe it component by component with an AWS stack. It is not an executable migration; it is the map of what disappears, what appears and where the lock-in shows up.

The network plane. Edge LB and WAF: AWS WAF + CloudFront. Ingress to the cluster: AWS Load Balancer Controller over EKS. What was Cilium BGP + RKE2 is replaced by EKS with VPC CNI (or Cilium on EKS, which is possible). The conceptual equivalent of Tetragon is Amazon GuardDuty for EKS + optional Falco. Moderate lock-in: network control couples to VPC.

The chat gateway and auth. What was a bespoke API gateway with JWT verification materialises as Amazon API Gateway + Amazon Cognito (or IAM Identity Center if it is B2B). The gateway’s AI-aware routing is covered with Bedrock + per-customer tags, or with AWS API Gateway custom authorizers invoking a Lambda for tenant resolution. High lock-in on the identity layer if Cognito is chosen.

The inference engine. Three different options, with a clear trade-off.

• Bedrock with a managed model (Claude / Llama / Mistral): all the operations of vLLM, K8s Operators, KV cache and disaggregated serving disappear. You move to Provisioned Throughput for a latency guarantee. You gain time-to-market; you lose control over prefill/decode, over custom LoRA adapters (Bedrock accepts Bedrock-managed fine-tunes but not arbitrary LoRAs), and you enter model lock-in (moving from Claude to Llama means changing API).
• SageMaker Endpoints with your own vLLM container: you keep vLLM and its optimisations, but K8s disappears and SageMaker replaces it as the orchestration plane. Inference Components allow you to densify multiple adapters. KV cache, prefix caching and LoRA hot-swap work the same. Moderate lock-in on the SageMaker SDK and on the Inference Components format.
• EKS with vLLM (the minimalist option): basically the OSS stack but with EKS instead of RKE2 and EBS/EFS instead of Ceph. Low lock-in, limited benefit from the cloud.

Data layer. The corpus moves to S3 with versioning, the embeddings to Amazon OpenSearch Service or to Aurora pgvector. The radical managed option is Bedrock Knowledge Bases: you upload documents to S3, it indexes them, it exposes a retrieval API. We remove Qdrant, we remove manual embedding pipelines, we remove part of Kafka + Flink. But control over custom reranking, fine-grained per-chunk ACL and the possibility of re-embedding with a new encoder of your own disappears. Bedrock KB uses the Titan or Cohere embedders available in Bedrock, and changing them means changing the whole index. ENS compliance: you have to validate that the buckets and the index live in EU regions and that the embedding model does too.

Stream + CDC. Kafka + Debezium is replaced by MSK + MSK Connect or by Kinesis + DMS. Schema Registry: Glue Schema Registry. The events remain functionally equivalent. Moderate lock-in if you go to Kinesis (Kinesis is not Kafka), low if you go to MSK (Kafka compatibility).

Data versioning. Here the gap is clear. S3 Versioning + Lake Formation + Glue Catalog is not DVC. To preserve the discipline of the previous post, (dataset_id, dataset_version, sha256_hash) propagated as an input artefact to the trainer, you can keep DVC over S3 (DVC works perfectly with S3 as a remote) or accept the limitation and register lineage manually in SageMaker Lineage Tracking. The first option preserves the practice; the second accepts degradation.

Tune stage. The LoRA adapter customer_support_v7 is trained with SageMaker Training Jobs on ml.p5.48xlarge instances (8× H100), using a standard HuggingFace + PEFT container. MLflow managed by SageMaker or your own MLflow on EC2 covers tracking. Alternative: if the black box is acceptable, Bedrock Custom Models with a dataset in S3 produces a fine-tuned Bedrock model without instantiating a GPU manually, at the cost of not being able to inspect the run.

Eval stage. Promptfoo + RAGAS in CI runs the same over CodeBuild. Bedrock Model Evaluation replaces a good part of the automatic eval suite. Bedrock Guardrails replaces NeMo Guardrails + Presidio + LlamaGuard, with the loss of transparency noted earlier.

Deploy stage. If Bedrock was chosen as the engine, this stage vanishes: Bedrock serves. If SageMaker Endpoints + vLLM was chosen, KServe is replaced by SageMaker Operators (or KServe is kept over EKS). The AI Gateway that in OSS was Envoy AI Gateway or LiteLLM becomes API Gateway + Bedrock or API Gateway + Lambda + SageMaker.

Observe stage. The OTel Collector remains the standard. Traces to AWS X-Ray + CloudWatch Application Signals. Metrics to Amazon Managed Prometheus. Logs to CloudWatch Logs + optionally OpenSearch for search. Langfuse is hosted on ECS Fargate or EKS because the cloud has no native equivalent of integrated prompt + traces + scores. Drift: SageMaker Model Monitor replaces Evidently / NannyML. eBPF (Pixie / Hubble / Tetragon) has no direct equivalent in managed AWS, so Falco or installing Tetragon on EKS is still the route.

Retrain stage. SageMaker Pipelines orchestrates the quarterly cycle. SageMaker Ground Truth + A2I replace Argilla. The feedback_signals table in Postgres stays as it is (RDS Postgres) or moves to DynamoDB for large scales.

How much the lock-in weighs. The component with the highest lock-in is Bedrock + Bedrock Knowledge Bases + Bedrock Guardrails: leaving that requires rewriting the inference plane and reindexing the whole RAG. Next is the SageMaker SDK (Pipelines, Endpoints, Training), where leaving costs but means rewriting scripts, not data. Data in S3 is portable (S3 → MinIO with rclone works). OTel observability is portable at almost no cost if the collector is kept as the abstraction. The auth gateway is the other high lock-in point if it uses Cognito.

What you gain. A dramatic reduction in GPU infrastructure operations, K8s patching, CUDA driver management, prefill/decode sizing, Ceph / MinIO management. A very short start-up curve: one request served in less than a sprint versus several weeks of bring-up for the OSS stack. Explicit provider SLAs.

What you lose. Contractual data sovereignty (the data stays in EU regions if configured that way, but the operator is a US third party under the Cloud Act). Visibility of the whole stack (Bedrock is a black box from the model downwards). Roadmap independence (the decision to discontinue a model, raise prices or change guardrails is not controlled by the customer). Fine-grained cost-per-token optimisation (the levers are the ones the provider exposes). For ENS customers under a HIGH declaration or NIS2 essential category, several of these points are non-compliance, not preference.

Master table: the entire parallel catalogue

When to choose each side — the real decision

The right question is not “OSS or cloud?”. It is stage by stage.

The OSS side wins by default when there is:

• Data subject to ENS HIGH category, NIS2 essential sectors or equivalents (identifiable health data, regulated banking, critical infrastructure). Here provider traceability and the processing contract are not negotiable; using a service whose operator is subject to the Cloud Act, FISA 702 or equivalent compromises the legal basis.
• A requirement for auditable inspection of the model, the guardrails and the whole pipeline. If a regulator asks “how exactly does it detect PII?” and an answer that ends in open code is mandatory.
• Large volumes with stable workloads. Above a certain threshold of tokens per month, the cost of Bedrock / AOAI / Vertex diverges from the amortised cost of your own GPU cluster. The threshold depends on the workload, but it typically sits between 5 and 50 billion tokens per month for models in the Llama 70B range.
• Roadmap independence as a priority. The day the provider discontinues a model or raises the price by 40 %, the organisation has to be able to ignore it.

The hyperscaler side wins by default when there is:

• Critical time-to-market, an MVP in weeks. The operational weight of the OSS stack is too much for a project that has not yet proved product-market fit.
• A small team with no SREs / MLEs specialised in GPU inference. Operating KServe + vLLM + KV cache + multi-tenancy is not trivial; if the team cannot sustain it, hosting is the way.
• Variable workloads / unpredictable spikes. Bedrock on-demand and SageMaker serverless charge for what you use; your own cluster pays for the GPU whether it is busy or not.
• A need for specific proprietary models (Claude, GPT-4.1, Gemini Pro) with no acceptable OSS equivalent for the case.

Mixed stages are frequent and reasonable. In practice, a common pattern in 2026 is: data, observe and retrain in self-hosted OSS (lineage and sovereignty), tune in OSS on your own cluster, eval in OSS + managed guardrails depending on the safety profile, deploy managed for proprietary models and self-hosted for open models. The question to ask for each stage is: “if the provider raises prices by 50 % or discontinues a component tomorrow, how much does it cost to move it?”. The parallel catalogue in this post gives the answer for each box.

What we have not covered (yet)

Some pieces still deserve a post of their own:

• OpenAI / Anthropic API directly (not through Bedrock or AOAI): another level of managed, another contract.
• Serious hybrids: AWS Outposts, GCP Distributed Cloud, Azure Stack HCI / Azure Local, the hyperscaler in your own room.
• Per-tenant cost accounting compared OSS vs cloud: how the bill is built and where attribution breaks.
• A real migration OSS → cloud or cloud → OSS: steps, scripts, gotchas.
• Concrete European sovereignty: GAIA-X, EuroHPC, the European cloud offering (OVHcloud, Scaleway, IONOS, Aruba), compared with the big three for ENS / NIS2 cases.
• AWS Inferentia / Trainium, GCP TPU v6 Trillium, Azure Maia: in-house chips and how they change the cost / token calculation.

See also

• Anatomy of an LLM request in production — the forensic walk of a request through the six stages, the thread from which this post makes the vertical cut.
• The OSS catalogue for LLMOps in six stages: tool by tool — the zoom into the open source side of this post’s master table: a card of about 150 words per core OSS tool (vLLM, Langfuse, DVC, Qdrant, Airflow, NeMo Guardrails, Presidio…), licence and governance, a decision matrix per stage and a diagram of the connected OSS stack.
• The six-stage LLMOps pipeline — the master map that this catalogue complements.
• MLOps specific to LLMs in 2026 — general context on why LLMOps is not classic MLOps.
• Data versioning with DVC and lakeFS — the deep-dive on the Data cross-cutting piece.
• Prompt versioning with Langfuse and MLflow — the deep-dive on the Prompt cross-cutting piece.
• Continuous fine-tuning in production — Tune in detail.
• Evals: the layer after tracing — Eval in detail.
• Guardrails and safety in LLMs — the safety layer in detail.
• KV cache · PagedAttention · Disaggregated serving — Deploy from the inside.
• vLLM on Kubernetes · LLM operators on K8s · Multi-tenant GPU cluster — Deploy in operation.
• AgentSight LLM tracing · MCP observability with OTel · eBPF + drift — Observe in its layers.
• Retrain: closing the loop feedback → dataset → adapter — Retrain in detail.
• ISO/IEC 42001: the operations manual of the AI system — the lock-in and contractual sovereignty analysis in this post feeds directly into Annex A.10 (third parties and customer relationships) of the AIMS; the AI supplier register demands documenting exactly what is compared here column by column.

References

• vLLM documentation and vLLM Production Stack — the reference OSS inference engine.
• Amazon Bedrock documentation — AWS managed model catalogue, Knowledge Bases, Guardrails and Prompt Management.
• Amazon SageMaker AI — training, endpoints, pipelines, model monitoring.
• Google Vertex AI documentation — training, prediction, evaluation, model monitoring.
• Azure AI Foundry documentation — Microsoft’s unified plane for AI applications.
• Azure OpenAI Service documentation — OpenAI models hosted on Azure.
• OpenTelemetry GenAI semantic conventions — the standard that stitches observability across the OSS / cloud boundary.
• Langfuse documentation and Arize Phoenix — the reference OSS LLM observability tools.
• DVC and lakeFS — OSS data versioning.
• NeMo Guardrails — OSS safety + dialog policy.
• ENS (Esquema Nacional de Seguridad) and NIS2 (Network and Information Security Directive 2) — the compliance frameworks that have the last word in the OSS vs cloud choice for regulated EU customers.