Kubeflow on lo0 — Tech Blog

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

Mon, 31 Aug 2026 09:30:00 +0200

Second post in the platform and self-service run, after Backstage and before Knative. The question here is not what Kubeflow is, but which parts of Kubeflow still make sense once you have built half a platform on your own.

TL;DR

Kubeflow graduated in the CNCF on 17 August 2026, and that headline hides the important one: it is no longer a monolithic platform, it is an umbrella of independent subprojects, each with its own repository, its own version and its own life. That changes how you have to evaluate it entirely. The 2020 question, should I install Kubeflow?, has become the 2026 one, which piece of Kubeflow do I install, and at what cost?

For a platform that already has RKE2, vLLM, KServe, Flux and Volcano or Kueue, the honest answer is that Kubeflow as a whole duplicates almost everything you already have. KServe no longer lives under Kubeflow: it left, and since November 2025 it has been a CNCF project in its own right, so anyone serving models with KServe does not need Kubeflow to serve. The Trainer leans on Volcano and Kueue, it does not replace them. And the full manifest drags in Istio, Dex and cert-manager as hard dependencies that clash with what you already operate. The only thing that may be worth installing on its own is Pipelines, as an orchestrator for data and training flows, and perhaps the Trainer for its fine-tuning templates. The rest you either already have, or do not need.

The analogy: the hypermarket and the specialist shop

Kubeflow was born as a hypermarket: a single building with everything an ML team might need, from notebooks to the inference server, taking in pipelines and hyperparameter tuning along the way. It made sense in 2019, when assembling each piece separately was a project in itself and there were no standards.

By 2026 the hypermarket has been converted into a street of specialist shops. The butcher’s counter, KServe, went independent, opened its own premises and is doing better alone. The fishmonger’s, the Trainer, now depends on the central market next door, Volcano and Kueue, instead of having its own cold store. And the shared building still demands that you put in its wiring, its plumbing and its security system (Istio, cert-manager, Dex) even though you already have your own.

Anyone arriving today with a half-full fridge, which is the case for anyone with a platform already running, does not need the whole hypermarket. They need to know which shop to walk into for what they are missing, and to avoid buying twice what they already have at home.

What Kubeflow is in 2026

Kubeflow was created by Google in 2017 and graduated in the CNCF on 17 August 2026, reaching the top level alongside Kubernetes or Prometheus. The graduation figures are serious: more than 6,600 contributors from over a thousand organisations, and adopters such as Bloomberg, NVIDIA, Red Hat or Spotify.

The structural change is the one that matters for the decision. The full distribution, the Kubeflow Community Distribution, is numbered by year and month (the current one is 26.03, from March 2026, on a six-monthly cadence), but each component is by now a subproject with its own repository and its own cycle, usable independently. The distribution only packages them together. That means “installing Kubeflow” is no longer an atomic decision: it is a shopping list.

Component by component

Kubeflow Pipelines: the piece you may actually want

Pipelines orchestrates containerised ML workflows. They are written in Python with its SDK, which compiles to an intermediate YAML representation so they are portable, and underneath they run on Argo Workflows. It is the piece most likely to add value to an existing platform: a flow of “ingest, embeddings, fine-tuning, evaluation, deployment” is exactly what you need to operate RAG and adapters repeatably.

The small print has two parts. The first is that if you already use plain Argo Workflows, Pipelines adds a layer of SDK, interface and metadata on top of the same engine: it does not bring a new scheduler, it brings the typed component model and the runs interface. The second is that its metadata and lineage system, MLMD, is being removed from Pipelines itself, so it is not wise to build the platform’s traceability on that particular piece.

Kubeflow Trainer: it leans on your scheduler, it does not replace it

The old Training Operator, the one with PyTorchJob and TFJob, has been rewritten as Kubeflow Trainer, with a single TrainJob API that unifies all frameworks. Version 2.2 positions itself explicitly for distributed training and LLM fine-tuning, with support for PyTorch, DeepSpeed, HuggingFace and company.

A common expectation needs correcting here. The Trainer does not replace Volcano or Kueue: it leans on them. The TrainJob carries a podGroupPolicy field that automatically creates the Volcano PodGroup objects for gang scheduling, and it integrates with Kueue’s queues. If you already built that scheduler, as we saw in its post, the Trainer adds the TrainJob abstraction and some fine-tuning templates on top of what you already have, not a new scheduling layer. It may be worth it for the templates; not for the scheduler.

Katib: only half relevant

Katib does Kubernetes-native AutoML: hyperparameter optimisation with algorithms such as Bayesian optimisation, TPE or Hyperband, plus architecture search. In the LLM era it has repositioned itself, with official documentation for tuning fine-tuning hyperparameters and even for tuning RAG pipelines.

The practical reading is lukewarm. Architecture search is irrelevant for anyone doing LoRA and RAG. Hyperparameter optimisation can be useful for learning-rate or LoRA-rank sweeps, but it competes with doing it from the SDK of the QLoRA runbook itself or with Ray Tune, and it rarely justifies dragging in Katib for that alone.

KServe: the piece that is no longer in Kubeflow

This is the fact that settles half the evaluation. KServe left Kubeflow. It was born inside the project in 2019, was donated to the Linux Foundation in 2022, was renamed from KFServing to KServe and graduated from Kubeflow that same year, and since November 2025 it has been a CNCF incubating project in its own right. In the 2026 taxonomy it appears as an ecosystem project, external, not as a core component.

The consequence is direct: anyone already serving with KServe does not need Kubeflow to serve. And KServe is exactly where the LLM action is: its 0.15 release strengthened the vLLM backend and introduced an LLMInferenceService resource with disaggregated serving, prefix caching, per-variant autoscaling and OpenAI-compatible APIs. If your serving layer is vLLM on KServe, you already have that layer complete and Kubeflow adds nothing to it.

Model Registry, Notebooks and the rest

The Model Registry, now reorganised under the name Kubeflow Hub, is an index of models, versions and metadata. It stores pointers and states, not the bytes. Against MLflow, it lacks the experiment and metric tracking that is MLflow’s strength, to the point that the guides recommend combining them rather than substituting one for the other; and against an OCI registry, it does not package the binary, which still lives wherever you put it. It is still a young component, below version 1.0.

The Notebooks live in two versions, the stable 1 and a 2 redesigned on CRDs that is still in alpha. The Spark Operator remains in the core; Feast, by contrast, has left the core and is now an ecosystem project, a nuance worth being clear about before counting on it.

Multi-tenancy: powerful and tied to Istio

Isolation between teams is done with Profiles, a CRD that wraps a namespace and gives it RoleBindings, ServiceAccounts and Istio authorisation policies that validate an identity header derived from OIDC. It is a complete model, but it has an Achilles heel: it depends entirely on the Istio sidecar and on that header. If traffic bypasses the mesh or someone forges the header, isolation breaks, and all multi-tenant security ends up coupled to Istio, which clashes head-on if you already operate another mesh or no service mesh at all.

The cost of installing the whole thing

Here is the main reason not to install the full distribution on top of an existing platform. The 26.03 manifest drags in as hard dependencies: Istio, cert-manager, Dex for OIDC, OAuth2-Proxy, and Knative Serving and Eventing for KServe. It recommends 16 GB of RAM and 8 vCPU as a minimum, and the aggregate of all components comes to around 4.4 CPU cores and 12 GB of memory for the control plane alone.

The project itself acknowledges the problem. There is an open thread in its repository, bluntly titled “community feedback on Kubeflow’s complexity”, where it is admitted that Kubeflow practically forces a dedicated cluster because of the assumptions it makes about what is installed, that Istio and Dex should be swappable and not fixed dependencies, and that a lighter Helm that installs only what is needed is missing. For anyone already operating cert-manager, an ingress and GitOps with Flux, the full manifest does not coexist: it duplicates and tramples the base infrastructure.

Kubeflow against the alternatives

The field is wide and almost all the comparative literature is written by vendors with a product of their own, so it has to be read with tweezers. With that caution:

Plain Argo Workflows is the engine Pipelines uses underneath; if all you need is container graphs, Pipelines is overhead. MLflow is strong in experiment tracking and lightweight registry, installs in minutes, and does not orchestrate distributed training: it is complementary, and in fact we use it in the post on prompt versioning. Flyte is a typed orchestrator that many perceive as lighter to operate than full Kubeflow. And Ray on Kubernetes is the most serious competitor for LLM workloads, because with a single runtime it covers training, serving and tuning, overlapping at once with the Trainer, Katib and KServe.

Where Kubeflow wins is in breadth and in governance: a CNCF-graduated umbrella with adoption by large companies. Where it loses is in operational weight, in hard dependencies, and in the fact that its best pieces are already independent projects that do not require the rest.

The real fit with LLM workloads

The project has moved towards GenAI, and that has to be acknowledged: there is an SDK with LLM fine-tuning templates, the Trainer adds support for reinforcement post-training methods, and new agent-oriented pieces have appeared. It is not empty marketing, there is real work there.

But the question for a specific platform is not whether Kubeflow does LLM: it is which part I have not already duplicated. On a stack of RKE2, vLLM, KServe, Flux and Volcano or Kueue, the balance looks like this:

Component	Does it add anything new to your stack?
KServe	No: you already have it, and it is no longer Kubeflow’s
Trainer	Only the fine-tuning templates; you already have the scheduler
Katib	Marginal: LoRA HP sweeps, little more
Pipelines	Possibly yes, as a flow orchestrator, but it runs on Argo
Model Registry	Optional: competes with MLflow and with your OCI registry
Notebooks	Depends on what you use for notebooks today
Multi-tenancy	No, if you already isolate by namespace without tying everything to Istio

Operational traps and honest scepticism

Do not install the full manifest on a platform that already works. You are going to duplicate Istio, cert-manager and the rest, and fight a dependency war with no prize at the end. If something in Kubeflow interests you, install it as a standalone subproject.

KServe is not Kubeflow. It is the most common category error of 2026. If someone justifies building all of Kubeflow “to serve models”, you already know they have not checked that KServe left long ago and lives on its own.

The Trainer does not save you the scheduler. It needs Volcano or Kueue underneath. If you expected it to solve GPU gang scheduling by itself, it does not.

Multi-tenancy ties you to Istio. Before adopting Profiles, ask yourself whether you want security between teams to depend on one specific mesh. If you already isolate well by namespace and RBAC, you may not need that layer.

MLMD is on its way out. Do not build your pipelines’ traceability on the metadata system the project itself is retiring.

For an inference factory

The conclusion is comfortable to apply. Kubeflow in 2026 is not a yes-or-no decision, it is a menu you order from à la carte. On a platform that already serves with KServe, schedules with Volcano and deploys with Flux, the menu shrinks a great deal.

If you need to orchestrate data and training flows repeatably, and you do not want to build plain Argo Workflows with its ergonomics, Pipelines is the piece worth evaluating, installed standalone. If you do distributed fine-tuning frequently and want a uniform abstraction on top of your scheduler, the Trainer may save you some templates, knowing that it leans on the Volcano you already have. Everything else you either have, or a specialist piece covers better.

The expensive mistake would be installing the whole hypermarket to buy bread. Kubeflow stopped being that; treat it as what it is, a street of shops, and walk only into the one you are missing. The next piece in this run, Knative, is precisely one of those specialist shops, the one that decides whether switching off the GPU when nobody is using it pays off or costs you dear.

Sources

CNCF, CNCF announces Kubeflow’s graduation — https://www.cncf.io/announcements/2026/08/17/cncf-announces-kubeflows-graduation-solidifying-the-standard-for-cloud-native-ai-operations/
CNCF, Kubeflow unveils new cloud native innovations to supercharge AI — https://www.cncf.io/blog/2026/07/28/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
Kubeflow Docs, Introduction / components — https://www.kubeflow.org/docs/started/introduction/
Kubeflow Docs, Pipelines overview — https://www.kubeflow.org/docs/components/pipelines/overview/
Kubeflow Trainer Docs, Volcano gang scheduling and Kueue — https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/volcano.html
Kubeflow Docs, Migrating to Kubeflow Trainer v2 — https://www.kubeflow.org/docs/components/trainer/operator-guides/migration/
Kubeflow Docs, Katib overview — https://www.kubeflow.org/docs/components/katib/overview/
Kubeflow Docs, Hyperparameter optimization for LLM fine-tuning — https://www.kubeflow.org/docs/components/katib/user-guides/llm-hp-optimization/
Kubeflow Docs, Model Registry overview — https://www.kubeflow.org/docs/components/model-registry/overview/
Kubeflow Docs, Multi-tenancy design — https://www.kubeflow.org/docs/concepts/multi-tenancy/design/
GitHub, kubeflow/manifests (dependencias y recursos) — https://github.com/kubeflow/manifests
GitHub, kubeflow/manifests #2451: community feedback on complexity — https://github.com/kubeflow/manifests/issues/2451
InfoQ, Kubeflow expands AI capabilities as CNCF graduation nears — https://www.infoq.com/news/2026/08/kubeflow/

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

Sat, 23 May 2026 07:30:00 +0200

TL;DR

For each of the six LLMOps stages (Data, Tune, Eval, Deploy, Observe, Retrain) and the two cross-cutting components (prompt + data versioning), the open source ecosystem has canonical pieces that this blog has been citing again and again. This post gathers them in one place with cards of roughly 150 words per core tool: what it does, how it differs from the alternatives inside the same bucket, its licence and governance model, and a typical gotcha that you only learn in production. Plus more alternatives as bullets, a decision matrix per stage according to the case (small / large corpus, single tenant / multi-tenant…), a diagram of the connected OSS stack and a master table of licences / EE offerings. The intention: that the reader closes the post knowing what is available, which company maintains it, what gap each piece fills, and when to choose it. This is not opinion: it is a curated catalogue.

You are here: every stage, but by OSS column

This post shares a map with the two previous ones in the series, the six stages and the two cross-cutting pieces are all active, but it zooms in on the open source column.

The analogy: the electrician’s toolbox

A professional electrician arrives at an installation with a box organised into compartments. Nothing is improvised: for every type of cable there is a specific wire stripper, for every screw a screwdriver of the exact gauge, for every measurement a multimeter and a clamp meter, for every connection the right terminal block or connector. The difference between a professional electrician and a handyman is not that one knows more theory, the handyman has often read the manuals, it is that the right tool is within arm’s reach and he knows when to use each one. The day the specific stripper is missing, improvising with a box cutter breaks the insulation, leaves a badly terminated cable, and the panel comes back under warranty two months later.

The OSS LLMOps stack works the same way. For every canonical problem, versioning a dataset, indexing a corpus for retrieval, serving tokens with dynamic batching, propagating trace_id end-to-end, managing prompts with a production label, orchestrating retraining pipelines, there is a canonical piece of the open source ecosystem that solves it, maintained by a serious community or foundation, with a clear licence and a well documented gotcha. The consultant who knows which tool to use for each job builds a robust system in weeks; the one who improvises with “whatever the team already knows” pays later in operations, usually once the system is carrying real load and any replacement is expensive.

This post opens the toolbox and shows each card. It is not a user manual, that is what the deep-dive posts linked at the end are for; it is the curated catalogue.

Diagram of the connected reference OSS stack

The catalogue makes sense once you see how the pieces connect into a single coherent architecture, the one this blog has been describing throughout the series. The boxes do not float; they talk to each other over stable contracts (HTTP, gRPC, OTel, Kafka, S3/MinIO API).

Solid arrows mark data / control flow; the dashed blue ones are OTel traces. The K8s plane holds everything up. The control plane at the bottom is where the retraining pipelines, the evals in CI, the versioned prompts and the lineage live. The data plane on the left feeds both the serving side (RAG, configs) and the control plane (datasets, lineage). The observability plane receives from serving and from everything else.

Now we go stage by stage. Each one opens with a context paragraph, then cards for the core tools (about 150 words each), bullets with relevant alternatives, and a specific decision matrix at the end.

Stage 1 — Data + cross-cutting Data versioning

The Data stage solves three distinct problems that beginners confuse: versioning datasets (so that (dataset_id, version, hash) exists and propagates), storing and serving the operational corpus (object store + vector index + structured text), and moving it between systems with CDC and stable schemas. Covered in detail in the posts on data versioning with DVC and lakeFS, PostgreSQL + Qdrant in ingestion and RAG over Kafka.

DVC (Data Version Control)

DVC puts datasets under version control with the same discipline that git applies to code. The .dvc pointers live in git (plain text, about 200 bytes per dataset), the bulk content lives in a remote object store (S3, MinIO, Azure Blob, GCS). Each dvc add computes a SHA-256 hash of the dataset, uploads it to the remote and stores the pointer. The key line: the dataset_hash becomes the luggage tag that travels to the trainer, to experiment tracking and to lineage. The same dataset retrained twice produces the same hash, and therefore reproducible experiments. DVC integrates with MLflow and W&B as an input artefact. Gotcha: it works well for datasets that change by replacement (I swap train.jsonl for a new version) and worse for datasets with thousands of small files that change individually. For that case, combine it with lakeFS. Licence Apache 2.0, maintained by Iterative.ai since 2017. There is DVC Studio (managed) and dvc data (pure CLI) on different planes.

lakeFS

lakeFS brings git semantics (branch, commit, merge, rollback) to an entire S3/MinIO/ADLS bucket. Where DVC versions individual files as pointers in git, lakeFS versions the whole bucket: you can branch the corpus, ingest new data into the branch, validate that it passes checks (recall@10 over golden queries for embeddings, completeness for a tabular corpus), and only then merge into main. It is the piece that makes continuous RAG safe: the production corpus is always on main, updates are tested on branches. It offers hooks (pre-merge, pre-commit) that fire automatic validations, and time-travel to reproduce the state of the bucket at a past date. Gotcha: the manifest overhead on huge buckets (hundreds of millions of objects) deserves sizing; lakeFS keeps metadata in its own Postgres, not in the bucket. Licence Apache 2.0, maintained by Treeverse since 2020. Managed offering: lakeFS Cloud.

MinIO

MinIO is the S3-compatible object store that fills the “S3 on-premise” gap without surprises. API identical to S3 (the AWS SDKs work by pointing them at a different endpoint), its own CLI client (mc), erasure-coded mode for fault tolerance, bucket-to-bucket replication, encryption at rest. It is the base on which the other data plane components are built: DVC remote, lakeFS underlying storage, Postgres snapshots, MLflow artefacts, eval datasets, saved models, distributed KV cache fabric. In small deployments it runs single-node multi-disk; in serious ones, distributed clusters. Gotcha: the licence changed to AGPLv3 in 2021 (it was Apache 2.0 before), which means distributing software connected to MinIO obliges you to open the code that connects to it. For internal on-premise use this is not a problem; for a vendor packaging MinIO inside a commercial product, it is. Maintained by MinIO Inc. with a SUBNET enterprise offering and a community fork called AIStor launched in 2025.

Qdrant

Qdrant is the OSS vector database most aligned with this blog’s “RAG corpus per tenant with strict ACLs” pattern. Written in Rust, it exposes a REST + gRPC API, indexes with HNSW + scalar/binary quantisation to cut memory, supports efficient payload filtering (it is not post-filtering: it integrates the filter into the HNSW search), and allows collections isolated per tenant. For the multi-tenant chatbot scenario, Qdrant is where the tenant_<id>_kb_v3 collections with strict ACL live. It scales well horizontally (sharding by payload) and vertically (millions of chunks on a node with 64GB RAM). Gotcha: binary quantisation is aggressive, it cuts VRAM by 32× but degrades recall by 10-20 %; enabling it without re-tuning the threshold breaks retrieval silently. Licence Apache 2.0, maintained by Qdrant Solutions GmbH (Germany). There is Qdrant Cloud (managed) and EU-only support for ENS cases.

PostgreSQL + pgvector

Postgres 18 with the pgvector extension is the stack’s “hidden vector database”: when the corpus is small (under a million embeddings) and Postgres is already in production for operational data, running a separate Qdrant is expensive operations. pgvector adds a vector(dim) type, HNSW and IVF indexes, and the operators <->, <#>, <=> for cosine, L2 and dot product. Combined with tsvector (Postgres full-text search) it allows hybrid search, dense + sparse, in a single SQL query. Version 0.8 (2025) introduced halfvec and bit support to reduce size by 4×-8×. Gotcha: HNSW in pgvector consumes a fair amount of RAM to build the index (roughly 2× the size of the embeddings) and blocks inserts during the build; in production you build on a secondary, promote it, and discard the primary. Licence PostgreSQL License (permissive BSD-style) for both the core and pgvector. Maintained by the PostgreSQL Development Group plus pgvector by Andrew Kane + Crunchy Data + Neon.

Apache Kafka + Debezium

Kafka is the event bus where “everything that happens in the company is a stream” becomes real. For LLMOps in production it serves two functions: CDC from source systems (Debezium captures changes in Postgres / MySQL / MongoDB and publishes them as topics) and LLM event buffer (every request, every piece of feedback, every eval result ends up in a topic with the trace_id propagated). As the post on RAG over Kafka explains, the RAG corpus is kept fresh by capturing changes from the CMS / source system as CDC, running the embedding in Flink streaming, and ingesting into Qdrant continuously. Gotcha: a badly sized Kafka with long retention plus multi-client topics turns into a fast disk sink; measuring throughput per topic and key cardinality before production is mandatory. Kafka licence Apache 2.0 (ASF project); Debezium Apache 2.0 (project incubated by Red Hat). Kafka-compatible drop-in alternative: Redpanda (BSL, restricted commercial use).

Apache Flink (brief mention)

Flink processes streams with sub-second latency and exactly-once semantics. On the LLM plane it is used to: run embeddings in streaming (over CDC topics), aggregate online metrics, materialise features for retraining. Licence Apache 2.0, ASF. Common alternative: Spark Structured Streaming (also ASF, micro-batch latency).

More options for Data, mentioned on the blog:

Ceph — object store for large clusters with geo-distributed replication. LGPL/Apache licence, Red Hat / IBM.
Milvus — C++ vector database, an alternative to Qdrant; better for corpora of billions. Apache 2.0, Zilliz.
Karapace — Confluent-compatible OSS Schema Registry. Apache 2.0, Aiven.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage. Apache 2.0, Acryl Data / ASF / Collate respectively.
OpenLineage — cross-system lineage event standard. Apache 2.0, Linux Foundation AI&Data.

Decision matrix — Data:

If your case is	Choose
Corpus < 1M embeddings, you already have Postgres	pgvector (one component fewer)
Corpus 1M-100M, multi-tenant with ACL	Qdrant (integrated filtering, ACLs per collection)
Corpus > 100M, aggressive sharding	Milvus (scales linearly better into the billions)
Training datasets + experiment tracking	DVC over MinIO + MLflow integration
RAG corpus with controlled releases	lakeFS over MinIO + pre-merge hooks
You want both	DVC + lakeFS as complements (the blog’s recommendation)

Stage 2 — Tune

The Tune stage produces a new model_id, model_version, typically a LoRA adapter over a stable base, with lineage back to the dataset and experiment tracking to reproduce it. Detail in the post on continuous fine-tuning.

HuggingFace Transformers + PEFT

transformers is the canonical library for loading and training models of the decoder-only family (Llama, Mistral, Qwen, Gemma…) and encoder-decoder. peft (Parameter-Efficient Fine-Tuning) is the complement that adds declarative support for LoRA, QLoRA, IA3 and assorted adapters. Together they form the mandatory core of the OSS Tune stack: any higher-level framework (Axolotl, LLaMA-Factory) uses them underneath. PEFT allows you to train an adapter of about 280 MB (order of magnitude) instead of a full model of about 140 GB, with a functionally equivalent result on most style / domain adjustment tasks. Gotcha: PEFT with a badly configured target_modules trains an adapter that covers only Q and V of the attention, leaving out key, output proj and MLP. The result looks trained but performs poorly; adding target_modules=["all-linear"] fixes it (at the cost of a larger adapter). Licence Apache 2.0, maintained by Hugging Face SAS (a French company); open governance model with active external maintainers.

bitsandbytes

bitsandbytes implements weight quantisation to 8-bit and 4-bit with NF4 for models loaded with transformers. It reduces the 140 GB of Llama 3 70B FP16 to about 40 GB in NF4, allowing QLoRA training on a single H100 80GB. The trick is that the weights stay quantised in memory while the sensitive computations (attention, gradient updates in the adapter) are done in FP16/BF16 with on-the-fly dequantisation. Ideal for fine-tuning on limited hardware and for serving with vLLM when you want to cut VRAM. Gotcha: NF4 quantisation is lossy; on small models (< 7B) the quality degradation is noticeable. For production serving of models under 7B, INT8 is preferred (more memory, less loss) or FP8 if the hardware supports it (the H100 does). Licence MIT, maintained by Tim Dettmers (originally at U. Washington, now with support from Anthropic and HuggingFace).

MLflow Tracking

MLflow is the reference OSS experiment tracking tool: each trainer run records parameters (lr, batch size, epochs, target_modules), metrics (loss curves, eval scores), artefacts (model, tokeniser, configs) and, crucially, input artefacts (dataset_id, dataset_hash, parent_run). The model registry associates each model_version with a reproducible run_id. The line of continuity between Tune and Deploy passes through here: the deployment reads the model to serve from the registry, with its lineage made explicit. MLflow 2.x integrates MLflow Prompts (prompt registry) and MLflow Tracing (OTel-compatible spans), cutting the number of components needed. Gotcha: the default backend store is SQLite, which works for personal experiments and breaks on a shared cluster. In production: Postgres as the backend store + MinIO/S3 as the artefact store. Licence Apache 2.0, maintained by LF AI & Data (donated by Databricks in 2020).

Axolotl

Axolotl wraps transformers + PEFT + bitsandbytes + DeepSpeed + FSDP in a declarative YAML configuration: instead of writing a script of about 300 lines to configure a fine-tuning run, you define a config.yml with base model, dataset path, LoRA config and training hyperparameters, and run it in one line. It supports Llama, Mistral, Qwen, Gemma, Phi… loads. It keeps compatibility with the HuggingFace Hub to download models and datasets, and with MLflow / W&B for tracking. It is the convenience framework this blog cites when it talks about “productive fine-tuning without reinventing the wheel”. Gotcha: the pace of change in the community is fast; a config.yml that worked six months ago can break with a current version because of internal refactors. Pinning the exact Axolotl version in the environment mitigates this. Licence Apache 2.0, maintained by the OpenAccess AI Collective (community-driven). A very similar alternative, more widely used in China: LLaMA-Factory (Apache 2.0, Beihang U.).

Ray Train

Ray Train scales fine-tuning to multiple nodes by distributing workers across a Ray cluster. While DeepSpeed and FSDP are intra-job parallelism (several GPUs collaborating on one job), Ray Train is the orchestration plane that builds the cluster, launches workers, manages checkpoints, recovers from node failures, and integrates with Slurm or Kubernetes. For training runs beyond 8 GPUs on changing clusters, Ray Train avoids the operational burden of “manually launching N torchrun processes with NCCL”. It combines with MLflow for tracking. Gotcha: the Ray learning curve is real; for a single node with 4-8 GPUs, torchrun or Hugging Face Accelerate are simpler. Ray Train shines when there are N changing nodes. Licence Apache 2.0, maintained by Anyscale Inc. (commercial backer) + community. A more K8s-native alternative: Kubeflow Training Operator (Apache 2.0, LF AI & Data).

More options for Tune:

DeepSpeed — ZeRO parallelism in 3 stages, mixed precision, CPU/NVMe offload. MIT, Microsoft.
FSDP (Fully Sharded Data Parallel) — native PyTorch parallelism, an alternative to DeepSpeed. BSD, Meta.
LLaMA-Factory — equivalent to Axolotl with a focus on the Llama family. Apache 2.0, Beihang University.

Decision matrix — Tune:

If your case is	Choose
Fine-tune on 1 GPU with 24GB (RTX 4090)	QLoRA with bitsandbytes NF4 + Axolotl
Fine-tune on 1 H100 80GB, models < 13B	LoRA bf16 + Axolotl
Fine-tune on 4-8 GPUs, single node	transformers + PEFT + Accelerate + MLflow
Multi-node fine-tune on a K8s cluster	Kubeflow Training Operator or Ray Train
Reproducible tracking is mandatory	MLflow + DVC input artefact
You want the minimum viable	Axolotl + MLflow

Stage 3 — Eval + Guardrails

Eval validates candidates pre and post promotion against a golden set with operational metrics; Guardrails runs safety online. Detailed in the posts on evals and guardrails.

DeepEval

DeepEval is the “pytest-style” OSS eval suite: you define tests with assertions over faithfulness, answer relevancy, contextual precision, hallucination rate, summarisation quality… and run them in CI. Each metric is an evaluator: some rule-based, others LLM-as-judge with auditable prompts. The philosophy is “evals as unit tests”: parameterisable by dataset, failable in CI, integrable with GitHub Actions. Gotcha: LLM-as-judge metrics vary between judge model versions, so if the judge moves up a version, the thresholds lose their earlier statistical meaning. Explicit pinning of the judge model in config plus periodic threshold recalibration is mandatory discipline. Licence Apache 2.0, maintained by Confident AI (a company); parallel commercial SaaS offering. Comparable: TruLens (MIT, TruEra) and G-Eval (academic).

RAGAS (RAG Assessment)

RAGAS specialises in evaluating RAG pipelines. It defines four canonical metrics: faithfulness (the answer is supported by the retrieved chunks), answer relevancy (the answer addresses the query), context precision (the retrieved chunks are relevant), context recall (all the relevant chunks were retrieved). Each metric is computed with LLM-as-judge over a dataset of (query, context, expected answer). For a RAG system, RAGAS is the evaluator that measures whether retrieval is aligned with generation. It integrates with Langfuse and MLflow to store results. Gotcha: RAGAS works well with golden sets of under 1000 examples; over huge golden sets the cost of the LLM judge per evaluation explodes, and the practice is to sample. Licence Apache 2.0, maintained by Exploding Gradients (the authors’ company).

Promptfoo

Promptfoo is the declarative, CI-oriented evaluator: in promptfooconfig.yaml you define a set of prompts and a set of assertions (contains text X, does not contain Y, faithfulness > 0.8, judge approves…), point at a provider (OpenAI compatible, vLLM, Ollama…), and promptfoo eval runs the matrix of prompts × providers × assertions, returns a diff against the baseline and fails CI if anything regresses. It is the most “DevOps-friendly” piece of the evals ecosystem: it integrates trivially with GitHub Actions, GitLab CI or Jenkins. Gotcha: assertion thresholds have to be calibrated with real data; starting at a default > 0.5 produces false positives that erode the team’s trust. Calibrate after the first week. Licence MIT, maintained by Promptfoo, Inc. (a company); a commercial Promptfoo Cloud SaaS offering exists but the OSS version is complete.

NeMo Guardrails

NeMo Guardrails is NVIDIA’s framework for defining and enforcing policies in LLM systems through a DSL called Colang. It lets you express rules such as “if the user asks about topic X, answer with template Y” or “if the model tries to do Z, block it” in a conversational-script syntax rather than in Python. It runs as middleware between the app and the model: input rails (validate what comes in), output rails (validate what goes out), dialog rails (control the flow). Designed for complex multi-turn systems where the policies are nontrivial. Gotcha: Colang adds latency per turn (roughly 50-200 ms depending on the policy graph); for high-throughput conversational chat, dialog rails are disabled and only input + output remain. Licence Apache 2.0, maintained by NVIDIA.

Microsoft Presidio

Presidio is the most mature OSS PII (Personally Identifiable Information) detector in the ecosystem. It detects DNI, NIE, IBAN, phone numbers, emails, physical addresses, credit card numbers, personal names, dates of birth… with recognisers based on regex + NER (spaCy) + custom validators. It supports redaction (replace with placeholders), masking (asterisks) or deterministic anonymisation (repeatable hash). For ENS/NIS2 scenarios, it is the piece placed in front (on input) and behind (on output) the LLM to guarantee that no PII is processed or emitted. Gotcha: the built-in recognisers cover English well and the rest badly; for Spanish, Catalan and Basque you have to add custom recognisers, which is disciplined work but doable. Licence MIT, maintained by Microsoft.

More options for Eval:

Phoenix Arize OSS — combines tracing + evals, an alternative to Langfuse Evals. ELv2, Arize AI.
lm-eval-harness — academic suite with standard benchmarks (MMLU, HellaSwag…). MIT, EleutherAI.
HELM — holistic academic evals. Apache 2.0, Stanford CRFM.
Guardrails AI — a pythonic alternative to NeMo Guardrails. Apache 2.0, Guardrails AI Inc..
LlamaGuard / PromptGuard / ShieldGemma — safety models, not frameworks. Open weights, Meta / Google.

Decision matrix — Eval + Guardrails:

If your case is	Choose
Eval in CI, “pytest for LLMs” style	Promptfoo + GitHub Actions
Eval specific to a RAG pipeline	RAGAS + Langfuse datasets
General eval with custom metrics	DeepEval + MLflow dataset
Dialog policy with declarative rules	NeMo Guardrails (Colang)
PII redaction in/out only	Presidio (you do not need NeMo)
Open safety model in Spanish	LlamaGuard 3 or ShieldGemma

Stage 4 — Deploy

Deploy serves tokens to the user with predictable throughput and latency, adapter hot-swap and multi-tenancy where it applies. Covered in the posts on vLLM on K8s, LLM operators, multi-tenant cluster, KV cache, PagedAttention and disaggregated serving.

vLLM

vLLM is the reference OSS inference engine. It implements PagedAttention (KV cache paging in the style of virtual memory, avoiding fragmentation), continuous batching (requests join the batch as they arrive, rather than waiting for the next batch), prefix caching (common prefixes, system prompts, do not recompute the KV cache), LoRA hot-swap (--enable-lora allows loading and unloading adapters without restarting the engine), an OpenAI-compatible API, and disaggregated prefill/decode support since 2025. It covers almost everything from Llama 3 / Mistral / Qwen / DeepSeek. Gotcha: maximum throughput is only reached with --max-num-seqs and --gpu-memory-utilization tuned for the specific model and hardware; the defaults are conservative. The initial tuning session pays off: 2-3× throughput. Licence Apache 2.0, originated at UC Berkeley, today maintained by the vLLM Project / LF AI & Data plus a broad community (Red Hat, NVIDIA, AWS, IBM contribute). Serious alternatives in the same bucket: TGI (Apache 2.0, Hugging Face), SGLang (Apache 2.0, LMSys), TensorRT-LLM (Apache 2.0, NVIDIA, requires conversion).

KServe

KServe is the Kubernetes operator for serving ML models, LLMs included, in a declarative pattern: you define an InferenceService YAML with the model and predictor (which can be vLLM, TGI, Triton, or a custom container) and KServe handles scheduling onto GPU nodes, autoscaling (scale-to-zero included), traffic splitting for canary, and model registry integration. It is the layer that standardises “how a model is deployed on K8s” across multiple engines, instead of inventing engine-specific YAML. It supports multi-model with Inference Graphs (chaining preprocessor → model → postprocessor) and integrates with KEDA/Karpenter for GPU pool autoscaling. Gotcha: scale-to-zero on GPU works badly in practice because the warm-up (loading weights into VRAM) takes tens of seconds; minReplicas: 1 is better. Licence Apache 2.0, maintained by Kubeflow / LF AI & Data. Alternatives: KubeRay (Apache 2.0, Anyscale), llm-d (Apache 2.0, CNCF), KAITO (MIT, Microsoft Azure).

Triton Inference Server

Triton serves heterogeneous models on a single backend: LLMs (via the vLLM or TensorRT-LLM backend), traditional models (ONNX, TorchScript, TensorFlow), custom models. For systems that mix LLM inference with traditional classifiers, embedding encoders, reranking models, OCR and so on, Triton avoids having N different engines in N pods. It supports ensemble models (chaining models in a single request), dynamic batching, model versioning, model warmup. Gotcha: Triton is flexible but heavy to operate; for systems serving only LLMs, vLLM directly is simpler and more optimised. Triton shines when there is real heterogeneity. Licence BSD-3-Clause, maintained by NVIDIA.

Envoy AI Gateway

Envoy AI Gateway is the CNCF ecosystem’s “LLM-aware API gateway”. Built on Envoy Proxy, it adds knowledge of the OpenAI-compatible APIs (chat completions, embeddings, etc.), routing across multiple backends (local vLLM + OpenAI + Anthropic + Bedrock), token-based rate limiting (limits by tokens per minute, not by requests), intelligent retries, fallback between providers, and built-in OTel observability. It is the piece that makes “AI Gateway” real as an architectural category. Gotcha: integration with authentication (OIDC, JWT) is flexible but requires detailed Envoy configuration; an “out of the box” AI Gateway with no configuration produces an Envoy that passes everything through. Licence Apache 2.0, maintained by CNCF since the initial donation from Tetrate. Alternatives: LiteLLM Proxy (MIT, BerriAI), Portkey (MIT, Portkey AI), Kong AI Gateway (Apache 2.0 base + EE, Kong Inc.).

llama.cpp

llama.cpp serves LLMs on CPUs (and Apple Silicon, GPUs via Vulkan/Metal/CUDA) with very aggressive quantisation (GGUF format, down to 2-bit). It is the canonical option for inference on hardware without a dedicated GPU: edge devices, workstations, development machines. It covers everything from small models (Phi-3, Gemma 2B) to Llama 70B on hardware with enough RAM. Gotcha: latency on CPU is orders of magnitude worse than on a dedicated GPU, so it is useful for offline evals, drift checks and local development, not for production serving under real load. Licence MIT, maintained by Georgi Gerganov + community.

More options for Deploy:

TensorRT-LLM — maximum optimisation on NVIDIA Hopper/Ada. Apache 2.0, NVIDIA.
SGLang — good for workloads with structured generation and JSON. Apache 2.0, LMSys.
TGI — a mature alternative, focused on the HuggingFace ecosystem. Apache 2.0, HuggingFace.
NVIDIA Dynamo — multi-node disaggregated serving. Apache 2.0, NVIDIA.
llm-d — K8s operator specific to LLMs. Apache 2.0, CNCF.

Decision matrix — Deploy:

If your case is	Choose
Production serving on NVIDIA H100/A100	vLLM (the safe default)
Absolute throughput squeezing on Hopper	TensorRT-LLM + vLLM plugin or standalone
Edge / local dev without GPU	llama.cpp
Multi-model (LLM + classifiers + encoders)	Triton with the vLLM backend
Declarative K8s with autoscaling	KServe + vLLM as predictor
AI Gateway with token rate limiting	Envoy AI Gateway
Multi-node disaggregated GPU cluster	NVIDIA Dynamo over vLLM

Stage 5 — Observe

Observe propagates trace_id end-to-end, emits runtime metrics, runs an LLM judge over sampling and detects drift. Detailed in tracing with AgentSight, MCP observability with OTel and eBPF + drift.

OpenTelemetry Collector

The OTel Collector is the agent that receives traces, metrics and logs in OTel format (or in any other via receivers), processes them (filters, sampling, attribute enrichment, PII redaction), and routes them to one or several backends (Tempo, Jaeger, Prometheus, Loki, Langfuse…). It is the piece that decouples the apps from the observability backend: moving from Tempo to Jaeger means changing the Collector’s exporter, not the app. For LLMOps it matters especially because the OTel GenAI semantic conventions spec defines the attributes gen_ai.request.model, gen_ai.prompt.version, gen_ai.response.tokens and so on, which stitch the trace_id to the system’s lineage. Gotcha: the Collector configuration tends to grow; without discipline and periodic review it ends up as an 800-line YAML that nobody understands. Modularising with extensions helps. Licence Apache 2.0, maintained by the CNCF / OpenTelemetry Project.

Tempo (traces) + Jaeger

Grafana Tempo is the distributed trace backend optimised for cost: it uses an object store (S3/MinIO) instead of Elasticsearch, deduplicates by trace_id, and integrates natively with Grafana for visualisation. For LLMOps, where a real request generates 10-30 spans (gateway, prompt pull, RAG retrieval, prefill, decode N times, scoring), Tempo copes with high volumes at a reasonable cost. Jaeger is the more established CNCF alternative, better for cases under 100k traces per day, worse for native object store. Gotcha: Tempo has no traditional indexing; searches like “traces that took more than 5s and touched tenant X” require TraceQL + Grafana and are not as fast as Jaeger with Elasticsearch. For immediate ad-hoc diagnosis it is worth keeping a parallel Jaeger with aggressive sampling. Licences AGPL 3.0 (Tempo) and Apache 2.0 (Jaeger), maintained by Grafana Labs and CNCF respectively.

Prometheus + Grafana

Prometheus is the time-series metrics foundation of the ecosystem. Pull model (it scrapes /metrics endpoints), PromQL for queries, exporters for everything (Postgres, Kafka, NVIDIA GPU via dcgm-exporter, vLLM natively). Grafana visualises Prometheus + Tempo + Loki on a single plane. For LLMOps, the critical metrics are gpu_utilization, kv_cache_usage_pct, tokens_per_second, prefill_latency_p95, decode_latency_p95, queue_depth, aggregated by tenant. Gotcha: Prometheus is very good up to about 1M active series; above that it is worth moving to Thanos or Mimir for long retention and horizontal scalability. For the blog’s typical LLM cluster (4-8 H100), Prometheus alone is enough. Licences Apache 2.0 (Prometheus, CNCF) and AGPL 3.0 (Grafana 10+, Grafana Labs).

Langfuse

Langfuse is the LLM-specific OSS observability + prompt management tool. It captures spans with LLM semantic conventions (input, output, model, tokens, latency, score, user_id, session_id), visualises them as conversational traces (not just span trees), manages prompts versioned with a production label and allows curated datasets + evals from the same UI. For serious LLMOps, Langfuse fills the gap that neither Tempo nor Jaeger covers: a tracing UI designed to be LLM-first. Gotcha: Langfuse maintains its own store (Postgres + ClickHouse for high volume); on large clusters the operational side of ClickHouse deserves attention. To get started, Postgres alone copes. Licence MIT for the OSS core, EE Enterprise Edition with additional features (SSO, audit logs, advanced RBAC). Maintained by Langfuse GmbH (Berlin, German). There is Langfuse Cloud (SaaS).

Phoenix Arize OSS

Phoenix is Arize AI’s OSS for LLM observability + evals, an alternative to Langfuse with a different emphasis: more oriented to evaluation and visual debugging (embedding drift, cluster analysis), less to prompt management. A good pairing with Langfuse when you want a dual approach: Langfuse for “production conversational traces”, Phoenix for “exploratory investigation of model behaviour”. Gotcha: Phoenix duplicates functionality with Langfuse and with MLflow; having all three in production multiplies the operational burden. Pick one as the main tool and the others as complements. Licence Elastic License 2.0 (not strictly OSI), maintained by Arize AI.

Cilium Tetragon + Hubble

Tetragon (eBPF runtime security observer) and Hubble (eBPF network observer) are the low-level pieces that give real runtime visibility into the cluster: which processes run in which pods, what syscalls they make, what network connections they open, in real time. For ENS/NIS2 environments that demand “prove what ran in production”, Tetragon is the irrefutable audit layer: every process execution with its parent, its capabilities, its K8s context. Hubble visualises network flows by pod, namespace, service. Gotcha: the volume of events generated is high; without kernel-side filtering (which Tetragon supports with TracingPolicy), it saturates the observability plane fast. Discipline in policies. Licence Apache 2.0 for both, maintained by Cilium / CNCF / Isovalent.

Evidently AI

Evidently is the OSS library for drift detection: it compares input and output distributions between two time windows (training vs production, current week vs previous week), applies statistical tests (KS, PSI, Wasserstein, chi-square) and generates HTML reports. For LLMOps it detects when the distribution of prompts changes (new topics, new lengths, new languages) or when the model starts answering shorter / longer / differently. Gotcha: Evidently is oriented to tabular data and embeddings; for raw text it is best combined with an encoder embedder that produces vectors before applying the tests. Licence Apache 2.0, maintained by Evidently AI (a company). Alternatives: NannyML (Apache 2.0, NannyML BV), Alibi Detect (Apache 2.0, Seldon).

More options for Observe:

Loki — Prometheus-style logs backend for Grafana. AGPL 3.0, Grafana Labs.
Pixie — auto-instrumented eBPF observability. Apache 2.0, CNCF.

Decision matrix — Observe:

If your case is	Choose
Minimum viable stack	OTel Collector + Tempo + Prometheus + Grafana + Langfuse
Traces with strong ad-hoc search	Add Jaeger with aggressive sampling
ENS / NIS2 compliance runtime audit	Tetragon + Hubble + mandated retention
Exploratory investigation of the model	Phoenix Arize OSS alongside Langfuse
Statistical drift detection	Evidently over embeddings + inputs
Cluster beyond 1M Prometheus series	Mimir (Grafana Labs) or Thanos

Stage 6 — Retrain + cross-cutting pieces

Retrain closes the loop feedback → triage → enriched dataset → new adapter. Prompt versioning and data versioning stitch lineage across stages. Detailed in retrain, prompt versioning and data versioning.

Apache Airflow

Airflow is the most established OSS DAG scheduler. You define workflows as Python code (DAGs), each DAG with tasks (operators) that run according to declared dependencies plus a cron schedule. For retraining: a weekly DAG that extracts feedback from Postgres, triages it with an LLM-as-classifier, enriches the enriched dataset in DVC, launches the fine-tuning job on Kubernetes, runs evals against the golden set, and promotes if it passes the gates. There is a huge ecosystem of operators for everything (S3, Postgres, Kafka, Slack, K8s, Spark…). Gotcha: Airflow 2.x improved a great deal over the chaos of 1.x, but the scheduler is still a component that deserves operational attention (Postgres backend, executor pool, sidecar workers); for simple pipelines it is over-engineering. Licence Apache 2.0, maintained by the ASF.

Argo Workflows

Argo Workflows is the K8s-native equivalent of Airflow: each step is a container, the DAGs are defined as K8s YAML, the executor is Kubernetes itself. For environments where everything is K8s, Argo fits without an extra component to maintain. Long tasks (a six-hour fine-tuning run) execute as Pods that survive control plane failures. It integrates trivially with Kubeflow Pipelines (which is built on top). Gotcha: Argo’s YAML syntax is verbose; for complex DAGs, Argo feels less productive than Airflow in Python. Solutions: Hera (Python DSL for Argo, a DataBricks contribution) or Argo + custom CRDs. Licence Apache 2.0, maintained by CNCF.

Kubeflow Pipelines

Kubeflow Pipelines is the layer above Argo Workflows oriented specifically to ML: artefact tracking, experiment tracking, reusable pipeline templates, versioned components. Built on Argo, it adds the ML conceptual model (input artefact, output artefact, metrics) that raw Argo does not have. For cyclical retraining on a K8s cluster, it is the most “ML-ready” option in the OSS ecosystem. Gotcha: Kubeflow as a full suite is heavy (10+ components); many organisations install only Pipelines + Training Operator + Katib and skip Notebook Server / legacy KFServing. Licence Apache 2.0, maintained by CNCF / LF AI & Data.

Feast

Feast is the most widely used OSS feature store. It defines feature views over batch sources (BigQuery, Postgres, Parquet) and online sources (Redis, DynamoDB, Postgres with an extension), exposes a consistent API for read-during-training and read-during-inference (point-in-time correctness), and guarantees that the features of the model in production are the same ones it was trained with. For LLMOps where the model needs consistent user / session / context features (latest plan, tenure as a customer, recent tickets), Feast provides the discipline. Gotcha: for many pure LLM systems (a RAG chatbot with no complex features), Feast is over-engineering, Postgres is enough. When there are real features (recommendation, scoring, ranking), Feast shines. Licence Apache 2.0, maintained by LF AI & Data.

Argilla

Argilla is the OSS annotation + HiL (human-in-the-loop) platform most aligned with modern LLMOps. It creates annotation projects with templates (classification, ranking, span annotation, RLHF preference, free-form text), connects to HuggingFace datasets, and integrates with Langfuse to import traces from production as cases to annotate. It supports multiple annotators with reconciliation, kappa scoring and quality control. To enrich retrain datasets with cases from the “blunt tone” cluster of the Retrain post, Argilla is the front end. Gotcha: Argilla requires Elasticsearch for production performance; for small experiments SQLite is enough. Licence Apache 2.0, maintained by Argilla, Inc. (acquired by Hugging Face in 2024). Alternative: Label Studio (Apache 2.0, HumanSignal), more generalist, less LLM-first.

Langfuse Prompts + MLflow Prompt Registry

Langfuse Prompts manages prompts as versioned entities with labels (production, staging, experiment). The client reads the active prompt from Langfuse on the request path (with a local cache of a few seconds) and propagates prompt_id, prompt_version to the OTel span, exactly as the forensic post does. MLflow Prompt Registry does the same with a slightly different conceptual model (no labels-as-pointers; it uses stages, like the Models registry). Both are valid; the choice depends on which tracking tool is already in place. Gotcha (Langfuse): labels are mutable, so changing production points at another version with no explicit audit; prompts are better deployed via PR against the configs repo, not manually in the UI. Licences and governance covered above.

More options for Retrain + cross-cutting pieces:

Prefect — “modern” Python DAGs, an alternative to Airflow. Apache 2.0, Prefect Tech.
Dagster — DAGs with a strong focus on data assets. Apache 2.0, Dagster Labs.
Label Studio — generalist annotation. Apache 2.0, HumanSignal.
OpenLineage — cross-system lineage event standard. Apache 2.0, LF AI & Data.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage with a UI. Apache 2.0.

Decision matrix — Retrain + cross-cutting pieces:

If your case is	Choose
Simple pipelines with a catalogue of operators	Airflow
Everything is K8s, minimal components	Argo Workflows
ML pipelines with artefact tracking	Kubeflow Pipelines
HiL annotation for LLM retraining	Argilla + Langfuse integration
Features shared between training and inference	Feast
No complex features, only prompts + LLM	Skip Feast
Lightweight prompt registry	Langfuse Prompts
MLflow is already centralised	MLflow Prompt Registry

Master table: licence, governance and enterprise offering

Tool	Licence	Governance / maintainer	Commercial EE / SaaS
DVC	Apache 2.0	Iterative.ai	DVC Studio
lakeFS	Apache 2.0	Treeverse	lakeFS Cloud
MinIO	AGPL v3	MinIO Inc.	SUBNET / AIStor
Qdrant	Apache 2.0	Qdrant GmbH	Qdrant Cloud
pgvector	PostgreSQL License	Andrew Kane + community	— (built-in Postgres clouds)
PostgreSQL	PostgreSQL License	PostgreSQL Global Dev Group	several managed (Crunchy, Neon, Aiven, EDB)
Apache Kafka	Apache 2.0	ASF	Confluent Cloud
Debezium	Apache 2.0	Red Hat / ASF	Debezium Server / Confluent Connectors
Apache Flink	Apache 2.0	ASF	Ververica Platform, Aiven
HF Transformers	Apache 2.0	Hugging Face SAS	HF Inference Endpoints / Enterprise Hub
PEFT	Apache 2.0	Hugging Face SAS	— (part of the HF offering)
bitsandbytes	MIT	Tim Dettmers + community	—
MLflow	Apache 2.0	LF AI & Data	Databricks MLflow
Axolotl	Apache 2.0	OpenAccess AI Collective	—
Ray (Train)	Apache 2.0	Anyscale + community	Anyscale Platform
DeepSpeed	MIT	Microsoft	—
DeepEval	Apache 2.0	Confident AI	Confident AI SaaS
RAGAS	Apache 2.0	Exploding Gradients	—
Promptfoo	MIT	Promptfoo, Inc.	Promptfoo Cloud
NeMo Guardrails	Apache 2.0	NVIDIA	NeMo Microservices
Presidio	MIT	Microsoft	—
Phoenix (Arize)	Elastic v2	Arize AI	Arize Platform
vLLM	Apache 2.0	vLLM Project / LF AI & Data	several (Red Hat, AWS, IBM, NVIDIA)
TGI	Apache 2.0	Hugging Face SAS	HF Inference Endpoints
SGLang	Apache 2.0	LMSys + community	—
TensorRT-LLM	Apache 2.0	NVIDIA	NVIDIA AI Enterprise
llama.cpp	MIT	Georgi Gerganov + community	—
Triton Inference Server	BSD-3	NVIDIA	NVIDIA AI Enterprise
KServe	Apache 2.0	LF AI & Data (Kubeflow)	—
Envoy AI Gateway	Apache 2.0	CNCF / Tetrate	Tetrate Service Bridge
LiteLLM	MIT	BerriAI	LiteLLM Cloud
OpenTelemetry	Apache 2.0	CNCF	several vendors (Honeycomb, Datadog, Grafana)
Tempo	AGPL 3.0	Grafana Labs	Grafana Cloud Tempo
Jaeger	Apache 2.0	CNCF	—
Prometheus	Apache 2.0	CNCF	Grafana Cloud, AMP, GCP Managed Prom, Azure
Grafana	AGPL 3.0	Grafana Labs	Grafana Cloud, Grafana Enterprise
Loki	AGPL 3.0	Grafana Labs	Grafana Cloud Loki
Langfuse	MIT (core) / EE	Langfuse GmbH	Langfuse Cloud
Tetragon	Apache 2.0	Cilium / CNCF / Isovalent	Isovalent Enterprise
Hubble	Apache 2.0	Cilium / CNCF	Isovalent Enterprise
Evidently AI	Apache 2.0	Evidently AI	Evidently Cloud
Apache Airflow	Apache 2.0	ASF	Astronomer, MWAA, Cloud Composer
Argo Workflows	Apache 2.0	CNCF	—
Kubeflow Pipelines	Apache 2.0	CNCF / LF AI & Data	—
Feast	Apache 2.0	LF AI & Data	Tecton (commercial)
Argilla	Apache 2.0	Hugging Face	HF Hub features
OpenLineage	Apache 2.0	LF AI & Data	—
DataHub	Apache 2.0	Acryl Data	Acryl Cloud

The pattern to watch for when reading the table: AGPL 3.0 and Elastic v2 are the ones that add the most friction in companies with strict licensing policies (legal asks for a specific review). The Apache 2.0 ones pass compliance without argument. Those with an “EE Enterprise” tier or equivalent hide a decision: the OSS version is functionally complete for production, but team features (SSO, audit, advanced RBAC) live in the commercial version. For ENS customers under a HIGH declaration, the EE features (SSO with corporate SAML/OIDC, immutable audit logs) are usually mandatory, so it is worth knowing the price beforehand.

When to move from the “minimum stack” to the “full stack”

The whole catalogue can be intimidating. But you do not build all of it on day one. There is a reasonable order that this blog has been validating in posts throughout the series. The minimum viable stack that serves an LLM API with acceptable discipline:

Serving: vLLM on Kubernetes plus an Envoy AI Gateway in front.
Data: Postgres + pgvector (no Qdrant), MinIO for object store, no Kafka.
Tune: Axolotl + MLflow, no Ray Train.
Eval: Promptfoo in CI, no RAGAS and no judge in production.
Observe: OTel Collector + Prometheus + Grafana + Langfuse, no Phoenix and no Tetragon.
Retrain: feedback in Postgres + crontab scripts, no Airflow.
Versioning: prompts in Langfuse + datasets in DVC over MinIO, no lakeFS.

That is about 8-10 components and it serves a reasonable LLM system for a single tenant with moderate traffic. As the system grows, there are identifiable moments where adding each piece pays off:

Trigger	Component to add
Multi-tenant with isolated corpora	Qdrant (collections per tenant, ACL)
The corpus is refreshed often and periodically breaks	lakeFS (branches with pre-merge hooks)
The embedding pipeline needs streaming	Kafka + Debezium + Flink
Retraining moves from monthly to weekly	Airflow or Argo Workflows
Shared features appear (customer profile, scoring)	Feast
Annotation outgrows informal capacity	Argilla
RAG eval needs specific metrics	RAGAS + Langfuse datasets
ENS compliance demands runtime audit	Tetragon + Hubble
Drift is invisible and shows up late	Evidently
A single stack stops covering multi-model	Triton or KServe with several predictors
Multiple simultaneous multi-tenant adapters	vLLM Production Stack + a dedicated Operator

Each jump adds 1-2 components and is worth the cost only when the trigger is clear. Adding Kafka “just in case” when the corpus is updated once a month is net negative work.

What we have not covered (yet)

Some pieces still deserve a post of their own:

Schema Registry for LLM data and prompts (Confluent OSS, Karapace, JSON Schema Registry).
Catalogue + lineage in depth: DataHub vs Atlas vs OpenMetadata + OpenLineage taken seriously.
Federated learning over OSS (Flower, FedML) for scenarios where the data is not centralised.
OSS MCP Servers and their place in the stack as a tools / actions layer.
“Agentic” evals specific to multi-step systems with tool use.
Upgrade best practices for each component (vLLM every 6 weeks, Kafka major every 18 months, etc.).

References

vLLM · vLLM Production Stack · TGI · SGLang · TensorRT-LLM · llama.cpp — OSS inference engines.
Triton Inference Server · KServe · Envoy AI Gateway · LiteLLM — orchestration and AI gateway.
Qdrant · pgvector · Milvus — vector databases.
DVC · lakeFS · MinIO — versioning and object store.
Apache Kafka · Debezium · Apache Flink — streams and CDC.
Hugging Face Transformers · PEFT · bitsandbytes · Axolotl — fine-tuning.
MLflow · Ray Train · Kubeflow Training Operator — training orchestration.
DeepEval · RAGAS · Promptfoo · NeMo Guardrails · Presidio — evals and guardrails.
OpenTelemetry · OTel GenAI semconv · Tempo · Prometheus · Grafana · Loki — observability foundation.
Langfuse · Phoenix Arize · Evidently AI — LLM observability and drift.
Cilium · Tetragon · Hubble — eBPF runtime.
Apache Airflow · Argo Workflows · Kubeflow Pipelines · Feast · Argilla — orchestration + retrain + annotation.

Kubeflow on lo0 — Tech Blog

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

TL;DR

The analogy: the hypermarket and the specialist shop

What Kubeflow is in 2026

Component by component

Kubeflow Pipelines: the piece you may actually want

Kubeflow Trainer: it leans on your scheduler, it does not replace it

Katib: only half relevant

KServe: the piece that is no longer in Kubeflow

Model Registry, Notebooks and the rest

Multi-tenancy: powerful and tied to Istio

The cost of installing the whole thing

Kubeflow against the alternatives

The real fit with LLM workloads

Operational traps and honest scepticism

For an inference factory

See also

Sources

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

TL;DR

You are here: every stage, but by OSS column

The analogy: the electrician’s toolbox

Diagram of the connected reference OSS stack

Stage 1 — Data + cross-cutting Data versioning

DVC (Data Version Control)

lakeFS

MinIO

Qdrant

PostgreSQL + pgvector

Apache Kafka + Debezium

Apache Flink (brief mention)

Stage 2 — Tune

HuggingFace Transformers + PEFT

bitsandbytes

MLflow Tracking

Axolotl

Ray Train

Stage 3 — Eval + Guardrails

DeepEval

RAGAS (RAG Assessment)

Promptfoo

NeMo Guardrails

Microsoft Presidio

Stage 4 — Deploy

vLLM

KServe

Triton Inference Server

Envoy AI Gateway

llama.cpp

Stage 5 — Observe

OpenTelemetry Collector

Tempo (traces) + Jaeger

Prometheus + Grafana

Langfuse

Phoenix Arize OSS

Cilium Tetragon + Hubble

Evidently AI

Stage 6 — Retrain + cross-cutting pieces

Apache Airflow

Argo Workflows

Kubeflow Pipelines

Feast

Argilla

Langfuse Prompts + MLflow Prompt Registry

Master table: licence, governance and enterprise offering

When to move from the “minimum stack” to the “full stack”

What we have not covered (yet)

See also

References