Autoservicio on lo0 — Tech Blog

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

Mon, 31 Aug 2026 09:45:00 +0200

Third and last article in the batch about rebuilding the cloud on-premise. The general map placed the gaps for you and Crossplane covered the infrastructure control plane. Here comes the layer above, the one your developer touches every day: application-centric self-service by CLI.

TL;DR

Crossplane answers “give me a database”, from the infrastructure side. KubeVela and Score answer “deploy my application”, from the workload side, and they do it by CLI, which is precisely the friendly experience Crossplane deliberately does not give you. The two share an article because they cover the same layer from different angles. KubeVela is a complete delivery platform: it uses the OAM model, where your platform team writes some definitions in the CUE language once and the developer deploys their whole application with a fifteen-line file and a CLI (vela up, vela status, vela logs) that feels like Heroku on the outside and is GitOps on the inside. Score is more modest and more portable: a workload specification in a single file that gets translated by CLI into Docker Compose locally and into Kubernetes in production, without running anything in the cluster.

The honest part up front is the limits. Neither of the two appears in the “Adopt” section of the CNCF technology radar from early 2026, where Helm, Backstage and Argo CD do: they are niche against the mainstream. KubeVela version 1.11 is still in alpha, so the stable one is 1.10; its community is concentrated, with around 33 people contributing in 2025. Score is young, its Kubernetes implementation has few stars and its sponsor, Humanitec, was acquired by Akamai. And KubeVela’s power lives in CUE, an uncommon language that is hard to debug. The abstraction is real, but when it breaks, your developer ends up looking at Kubernetes YAML all the same.

The analogy: the order form against the shopping list

Deploying on raw Kubernetes is going to the wholesaler with a technical shopping list: you have to know that you want a Deployment, a Service, a HorizontalPodAutoscaler, an Ingress, a ConfigMap and a ResourceQuota, and how each one gets filled in. It works if you know the warehouse. But your data scientist, who just wants to serve their model, should not have to learn the warehouse.

KubeVela is an order form. Instead of the technical list, you tick “I want to serve this application, with these replicas, with this gateway”, and behind it somebody who does know the warehouse (your platform team) has defined exactly what gets ordered for that to work. The developer fills in the form; the translation into the technical list was done by somebody else, once, for everyone.

Score is the same form but designed so that it works for you in two different shops. You describe your order once and the same sheet serves the market next door (Docker Compose, your laptop) and the wholesaler (Kubernetes, production), without rewriting it. Whatever changes between one shop and the other is supplied by whoever translates, not by you. Both analogies point at the same thing: separating what the developer wants from how it materialises, so that they do not have to learn the warehouse.

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

KubeVela is a CNCF project at incubation level since February 2023. It was born as an evolution of the OAM runtime on Kubernetes, with initial contributions from Alibaba Cloud, Microsoft and Upbound, and it rests on the Open Application Model, the specification Alibaba and Microsoft co-created to describe applications independently of the infrastructure.

Here is a realism check on its community size. The official 2025 retrospective declares seven releases in the year, 124 commits and around 33 people contributing, on a little over 7,600 GitHub stars. It is not a project in decline, but it is a concentrated community, with a growth curve that has flattened compared to its early stage under Alibaba. And an important version warning: the stable branch is 1.10, with regular patches through 2026; 1.11 is still in alpha, so do not rely on its features as if they were available.

The OAM model in five pieces

Your whole application is described in a single YAML file of type Application, with four sections you need to know by name. The components are the artefact you deploy, be it an image, a Helm chart or a service; the traits are operational requirements you hang off each component, such as scaling, the gateway or storage; the policies are global application strategy, such as multi-cluster topology or SLOs; and the workflow is the step-by-step delivery process, which can include a manual approval pause or a notification.

The key to the model is that component and trait types are not fixed: they are programmable modules maintained by your platform team, expressed as ComponentDefinition and TraitDefinition. The developer only consumes a type and its properties; they never see the Kubernetes YAML underneath. That is exactly the “do not expose raw Kubernetes” this is all about: your platform operator writes the definition once, your data scientist writes fifteen lines of Application.

CUE, the power and the toll

The definitions are written in CUE, and here you have KubeVela’s strength and weakness at the same time. CUE lets you parameterise seriously: the definition declares which inputs it exposes to the developer and which Kubernetes resources it renders from them, with access to context variables such as the application name. The real flow starts from an existing YAML, which vela def init turns into a CUE skeleton; you edit it, validate it with vela def vet and publish it with vela def apply, and it becomes available to all your developers.

The toll is that CUE is an uncommon language, with few experts and hard to debug. The cost falls on your platform team, not on the developer, which is the correct distribution, but it is a real cost you have to sign up for before adopting. Nobody arrives already knowing CUE.

The vela CLI: Heroku on the outside, GitOps on the inside

Here is the answer to the question this blog has been dragging along about CLI provisioning. Day to day with KubeVela looks like Heroku: vela up deploys the application from local files, vela status shows you its state, vela ls lists the applications, vela logs follows the logs, vela exec runs a command inside the container without you knowing the pod name, and vela port-forward opens a tunnel for you. You even have vela dry-run, which renders the Kubernetes resources without applying them, so you can see what is going to happen.

The honest qualification is that it is “imperative on the surface, declarative underneath”. When you run vela up you are not executing an imperative command like heroku scale, you are applying a declarative Application object that a controller reconciles. The CLI imitates Heroku’s comfortable flow, but the underlying model is GitOps, which means it fits with your Flux stack through the corresponding addon, instead of replacing it.

Multi-cluster, which is what matters to you

If you have a platform with several sites (site01 and site02, in my case) the multi-cluster part is the deciding one. KubeVela orchestrates from a central cluster and only the already rendered resources reach the managed clusters, with centralised governance. A topology policy decides the destination by cluster list or by label selector (by region, for example); an override policy adjusts images, replicas or traits per cluster without touching the component; and the deployment step in the workflow distributes, with the option of a phased rollout: first to the local cluster, then a manual approval gate, and only then to production. It is the kind of governance you have to build on-premise and that the cloud served you ready-made.

VelaUX, the lightweight portal thrown in

KubeVela comes with an optional web console, VelaUX, which you switch on as an addon and which gives you a dashboard, project management, environments and application lifecycle. If you want CLI first it is optional, more useful for giving operators visibility and access control than as the main route. It is a lightweight portal “thrown in” that does not aim to compete with Backstage, but it can save you from setting one up if your needs are modest.

Score: describe the workload once

What it is and what it solves

Score is an open workload specification, agnostic of platform and environment. The idea, in its own formulation, is that you define the workload once and use a Score implementation to translate it to several platforms: Docker Compose, Kubernetes, Fly.io or Cloud Run. The problem it attacks is concrete and familiar: the divergence between what you run on your laptop with Docker Compose and what gets deployed in production with Kubernetes, two descriptions that drift apart and cause you the classic “it worked on my machine”.

In a score.yaml file the developer declares their containers, the service ports and the resources they need as dependencies (a database, a DNS record, a route), and the environment-specific parameters are injected at deploy time. Score entered the CNCF sandbox in July 2024. It was created and donated by Humanitec, a developer platform company, some eighteen months before it joined the foundation.

The two implementations

Score does nothing on its own: it is a specification that needs an implementation to translate. You have two official ones. score-compose generates Docker Compose for local development, with init and generate commands, and provisions on the order of fourteen resource types, from Postgres or Redis to S3 storage or queues. score-k8s generates Kubernetes manifests ready for kubectl apply, with flags for the image, the namespace and the provisioners. Humanitec’s commercial CLI, humctl, also supports Score natively.

The most important architectural fact is that Score runs nothing in the target cluster. It is purely CLI translation: it generates manifests for you and steps aside. There is no controller, no CRD, no reconciliation. That is at once its virtue (it is lightweight, portable and without runtime lock-in) and its limit (it neither orchestrates nor deploys, it only translates).

Score against KubeVela and Helm

Score’s own official positioning clarifies the layers, and I will pin it down for you because they are easy to confuse. Score is the workload specification layer, developer-centric: it describes a workload and its dependencies and gets translated to N platforms by CLI, but it does not orchestrate or deploy. KubeVela is a complete continuous delivery platform, application-centric: it orchestrates multiple components, applies policies and workflows, does multi-cluster, and for that it needs its controller in the cluster; it is, in its own words, far richer in features than Score. Helm is the packaging and rendering of manifests, without the developer-centric abstraction or the portability to Compose.

The clean synthesis is that they are complementary layers, not substitutes: Score generates, Helm or the manifests package, KubeVela orchestrates and deploys. If they are presented to you as rivals, somebody has confused the layers.

Where each one fits: the cross-cutting comparison

With Crossplane already in hand from the previous article, the comparison of the three self-service pieces sorts itself by the question each one answers.

Score and KubeVela answer at the application level: the workload a developer deploys, “deploy this app”. Crossplane answers at the infrastructure level: the resource a developer asks for, “give me a managed Postgres”. And Backstage answers at the portal level: the web catalogue and the golden paths, the discovery layer that neither deploys nor provisions by itself, but orchestrates the ones below.

The interesting part is that they combine. A score.yaml can declare its resources and delegate the provisioning of those resources to Crossplane through a provisioner, while KubeVela orchestrates the application deployment and Backstage acts as the façade. It is not an either-or; it is a stack where each layer does its job. If your team is small, you pick a single piece and that is that; the complete stack is for mature platforms.

The fit with inference: a pattern, not an official recipe

Just as with Crossplane, honesty is called for here: there is no official, maintained KServe definition in KubeVela’s default catalogue, nor an inference resource type in Score. What follows is a design pattern with pieces confirmed separately, not a documented recipe.

The KubeVela pattern would be that your platform operator defines once, in CUE, a ComponentDefinition called for example inference-service, which underneath renders a KServe inference service with vLLM runtime, with the GPU limits, the namespace quotas and the autoscaling already wired in. Your data scientist then writes a fifteen-line Application with the inference-service type and its properties (the model, the replicas), without touching KServe, without knowing the GPU quotas or the runtime class, and operates with vela up, vela status and vela logs. Multi-site is solved for you by the topology policy, which decides which GPU cluster it lands on, and the override policy, which adjusts replicas or model per site, with a manual approval in the workflow before production.

The Score alternative would be a score.yaml with a resource of type inference-endpoint that score-k8s, with a custom provisioner, emits as a KServe inference service. More portable, but today it requires that provisioner of your own, because KServe is not a default type. In both cases, the abstraction layer rests on pieces that are already standard (KServe moved to CNCF incubation at the end of 2025 and vLLM is its reference generative runtime), but the inference abstraction is built by you. Take it as a proposal, not as a proven case.

A note for you: in 2026 KServe introduced a type dedicated to large models, LLMInferenceService, still in alpha and with vLLM as its runtime, which is the brick a KubeVela definition or a Score provisioner would render underneath. And at the infrastructure layer, not this one, the Crossplane team published Modelplane, an inference control plane made of compositions that serves vLLM directly. Neither of the two is a KubeVela or a Score type, but both confirm for you that the brick underneath is standardising, which is what makes putting a self-service façade on top viable.

Caveats, no frills

This is the section that separates the article from the brochure, and with these two projects you need it especially.

Adoption is niche, and there is a hard figure. The CNCF technology radar from early 2026, covering more than 400 developers, places Helm, Backstage and Argo CD in “Adopt”. Neither KubeVela, nor Score, nor Crossplane get a mention. They are niche tools against the mainstream, and choosing them means betting on an adoption curve that may not take off.

Score is young and its sponsor changed hands. Its Kubernetes implementation has few stars, the specification is at 0.x versions, and Humanitec, which sponsors it, was acquired by Akamai in 2025, so you have to watch the continuity of the investment in the open source part.

KubeVela has a concentrated community and v1.11 in alpha. The 33 people contributing in 2025 and the flat star curve are a sign of a plateau. Do not build on 1.11 features until it is stable.

The CUE curve is real. KubeVela’s power lives in CUE definitions, and CUE is hard. The cost falls on the platform, but it exists.

And the usual risk with abstractions: the leak. Both KubeVela and Score add an indirection over Kubernetes and KServe. When something fails underneath, the abstraction cracks and your developer ends up looking at the KServe YAML anyway, plus lock-in on your own definitions. An abstraction that does not hold up under failure hands you back exactly the complexity it promised to hide.

For your sovereign inference factory

The practical decision sorts itself by the profile of your users. If your people live in the CLI and think in applications, KubeVela gives you the Heroku-style experience on your own iron, with real multi-cluster, in exchange for your platform team learning CUE. If what hurts most is the divergence between local and production, Score solves that with the bare minimum, without putting one more runtime in the cluster. And if you want both plus a portal, the Score, Crossplane, KubeVela and Backstage stack exists, but it is for a platform that already serves many teams, not for the start.

The sobriety criterion is the same as throughout this batch: these tools plug a real gap left by the cloud you walked away from, but each one is one more system to operate. For two teams who know each other, good Helm and a tidy GitOps repository do the job without CUE or definitions of your own. Application-centric self-service starts paying you back when the number of people deploying exceeds the number of people who know Kubernetes, which is precisely the moment an inference factory stops being a project and starts being a platform.

That closes the batch. The map drew the gaps of leaving the cloud, Crossplane covered the infrastructure control plane, and KubeVela with Score cover the face the developer touches. Your own house now looks quite a lot like the hotel, and the data stayed inside.

Sources

KubeVela, State of KubeVela 2025 — https://kubevela.io/blog/2025/12/20/state-of-kubevela-2025/
CNCF, KubeVela brings software delivery control plane capabilities to the CNCF incubator — https://www.cncf.io/blog/2023/02/27/kubevela-brings-software-delivery-control-plane-capabilities-to-cncf-incubator/
KubeVela Docs, Core Concept — https://kubevela.io/docs/getting-started/core-concept/
KubeVela Docs, vela CLI reference — https://kubevela.io/docs/cli/vela/
KubeVela Docs, Custom Component with CUE — https://kubevela.io/docs/platform-engineers/components/custom-component/
KubeVela Docs, Multi-cluster delivery — https://kubevela.io/docs/case-studies/multi-cluster/
KubeVela Docs, VelaUX addon — https://kubevela.io/docs/reference/addons/velaux/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Score Docs, Overview — https://docs.score.dev/docs/
Score, Score vs the Open Application Model and KubeVela — https://score.dev/blog/score-vs-open-application-model-kubevela/
GitHub, score-compose — https://github.com/score-spec/score-compose
GitHub, score-k8s — https://github.com/score-spec/score-k8s
CNCF, KServe becomes a CNCF Incubating Project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
CNCF y SlashData, Technology Radar Q1 2026: Platform Engineering — https://www.cncf.io/announcements/2026/03/24/cncf-and-slashdata-report-finds-platform-engineering-tools-maturing-as-organizations-prepare-for-ai-driven-infrastructure/

Crossplane: the control plane that turns your cluster into a private cloud API

Mon, 31 Aug 2026 09:30:00 +0200

Second article in the batch about rebuilding the cloud experience on-premise. The general map placed the gaps for you; this one goes into the provisioning control plane, that API which on-premise does not come for free. The third moves up a layer, to application-centric self-service with KubeVela and Score.

TL;DR

Crossplane solves the “open a ticket and wait” gap for you. In the cloud you ask for a database through an API and it appears with permissions, backups and monitoring; on-premise you do that provisioning by hand or with scripts. Crossplane brings the Kubernetes model of declarative APIs to infrastructure: your platform team defines an API of its own (a database, an inference endpoint) and the developer consumes it with a kubectl apply of a few fields, while controllers inside the cluster reconcile the desired state against reality and correct drift continuously. It graduated from the CNCF in November 2025, with more than three thousand contributors and adopters such as Nike, SAP, IBM or NASA’s science cloud.

The difference from Terraform is one of nature, not of features: Terraform applies a plan on demand and leaves; Crossplane is a control plane that lives inside the cluster and never stops reconciling. That gives you drift self-correction and native GitOps, and in exchange it takes away the safety net of plan and forces you to operate the control plane itself, with providers that create hundreds of resource definitions and state that lives in etcd. And a warning about something awkward up front: Crossplane is not an imperative aws ... style CLI, it is declarative first. On the fit with GPU inference, which until recently was only a pattern with no public case, you now have a real reference, Modelplane, from the Crossplane team itself, although at version 0.1 and composing vLLM directly without going through KServe. I go into the detail at the end.

The analogy: the thermostat against the light switch

Think of it this way: Terraform is a light switch. You flip it, the light comes on, and it stays there until you touch it again. If somebody else changes it, the light obeys whoever touched it last, and you do not find out until you go and look. It is provisioning by one-off action: you run it, it applies, it finishes.

Crossplane is a thermostat. You do not tell it “turn the heating on”, you tell it “I want twenty-one degrees”, and from then on a mechanism watches the temperature and acts on its own, continuously, to hold it. If somebody opens a window and the thermometer drops, the thermostat reacts without you intervening. That is continuous reconciliation: you declare the desired state once and a controller takes care of reality not straying from it.

The analogy also warns you about the cost. A thermostat is more convenient than a switch, but it is one more device you have to install, power and fix when it fails, and if you program it badly it adjusts the temperature to something you did not want, without asking. A switch almost never breaks and you always know what state it is in. Crossplane gives you the convenience of the thermostat in exchange for you operating the thermostat.

What Crossplane is and what state it reaches 2026 in

Crossplane was born at Upbound in 2018 and followed the CNCF maturity path without shortcuts: sandbox in 2020, incubation in 2021 and graduation on 6 November 2025. Graduation matters because it is the seal the CNCF reserves for projects it considers production-ready, the same level as Kubernetes or Prometheus. The figures that came with the announcement are solid: more than three thousand contributors from more than 450 organisations, more than a thousand pull request authors (which puts it in the top 10 % of the foundation’s projects by that metric), more than a hundred releases and two completed security audits. Among the more than seventy public adopters you have Nike, Nokia, Grafana, NASA’s science cloud, SAP and IBM.

The project’s official description is “the cloud native control plane”, and that phrase sums up exactly what it does. The main vendor behind it is still Upbound, which at graduation framed the project’s future towards “AI-native” infrastructure, a point I come back to when talking about inference, with the caveat that it is a vendor’s positioning, not a proven case.

The release cadence is quarterly, with a nine-month support window and three actively maintained versions at any time. As of writing, the v2 branch is the current one, with v1.20 as the last close of the v1 branch.

The v2 turn (which changes how you think about it)

Crossplane 2.0 reached general availability in August 2025 and brings four structural changes you need to know about, because a good deal of the old material describes a model that is no longer the recommended one.

The first is that resources are now namespaced by default. Both composite and managed resources, which used to be cluster-scoped, now live in a namespace, which enables fine-grained access control and natural multi-tenancy, exactly what you need in a sovereign platform with several teams.

The second is that a composition can include any Kubernetes resource, not only Crossplane resources. Previously a composition was limited to Crossplane’s own managed resources; now it can include Deployments, Services or third-party CRDs, so you can compose abstractions mixing application and infrastructure in a single object. This change is what makes the inference pattern you will see viable.

The third is that Claims disappear. In the v1 model there was a duality between the composite resource, cluster-scoped, and the Claim, its namespaced face which the developer consumed. Now that the composite resource is itself namespaced, that duality is redundant: the developer consumes the composite resource with no intermediary. If you come across documentation talking about Claims, it belongs to the earlier model.

The fourth is a new type, Operations, for one-off or scheduled operational tasks that run a pipeline of functions to completion, like a Kubernetes Job but inside the Crossplane model.

There is one removal that deserves its own paragraph, because it is the most surprising: in v2 native patch-and-transform was withdrawn, the classic patching mechanism of compositions. It was not relegated, it was removed. Everything now goes through composition functions, and I get to that in a moment. Migration from v1 to v2 is mostly non-breaking and the v1 APIs still work, so adopting v2 is optional, but the new model is the one you have to learn.

The model, piece by piece

Crossplane has dense jargon and I will take it apart calmly, because once the pieces are placed the whole thing is coherent.

A provider is a package that installs controllers and resource definitions to talk to an external API, be it AWS, GCP, a remote Kubernetes cluster, Helm or Terraform. It is installed declaratively, by applying a Provider object. The providers for the big cloud providers are generated with Upjet, a framework that produces the code from the Terraform providers, which explains why there are so many of them and also why some are enormous.

A managed resource is the one-to-one representation, inside Kubernetes, of a resource from the external provider: a bucket, an instance, a database. It is the unit the controller reconciles. When you create a managed resource of type bucket, the provider’s controller creates the real bucket and then watches that it keeps existing with the configuration you declared.

On those bricks you build the abstraction, which is where the real value is, with three concepts:

The CompositeResourceDefinition, or XRD, defines the schema of your own API: which fields the user can ask for. It is what turns “a database” into a resource type with its parameters.

The Composition is the template and the logic that, given one of those resources, materialises the N managed resources needed underneath. The documentation says it bluntly: a composition is a pipeline of composition functions.

The Composite Resource, or XR, is the concrete instance: the object the developer creates with kubectl apply, with the few fields the XRD exposes.

The complete pattern is this: your platform operator publishes the abstraction (the XRD plus the Composition, which say “this is what a database means here”), and the developer only applies a composite resource with four fields. How it is built underneath stops mattering to them, just as in the cloud you do not care how AWS assembles your managed database.

Composition functions: the real logic

The composition mechanism in v2, and the only one, is composition functions. A composition is a pipeline of functions that Crossplane invokes over gRPC; each function receives the observed state, the desired state, its input and a shared context, and chains its output to the next one. The YAML in the examples is illustrative, but underneath it is not processed as YAML: it is executed as functions.

The available functions cover several styles. You have function-patch-and-transform, which reimplements the old model now as just another function. There is function-go-templating for Helm-style templates, function-kcl for the KCL language, function-python for logic in Python and function-cue for CUE.

The reason for the change is what you want to understand. Classic patch-and-transform had no loops or conditionals, so an operation as simple as transforming an array of strings turned into dozens of lines of patching, and the community went as far as calling it outright bad. Functions give you full programming logic (loops, conditionals, external calls) in a real language. The honest flip side is that the complexity does not disappear: it shifts from “writing verbose patches” to “writing and operating functions”, which is a different kind of work, not less work.

Crossplane against Terraform, OpenTofu and Pulumi

This is the comparison everybody makes, and I will make it properly because the difference is one of category. According to Pulumi’s own documentation, which compares honestly, Crossplane is a declarative control plane with continuous reconciliation inside Kubernetes, while Terraform and Pulumi are infrastructure as code executed one-off, client-side and on demand.

The practical consequences line up along a few axes. Faced with configuration drift, Crossplane corrects it continuously and automatically, while Terraform and Pulumi only detect it when you run a refresh. Crossplane needs a cluster to run on, because it is controllers; Terraform and Pulumi do not. Crossplane is GitOps-native, applied with kubectl and fitting with Flux without an adapter; Pulumi needs its Kubernetes operator for the same thing. And as for language, Crossplane uses YAML and CRDs with logic in the functions, while Pulumi uses general-purpose programming languages.

The point most often forgotten is that you do not have to choose one or the other. There is a Terraform provider (and an OpenTofu one, recommended for new versions because the Terraform one was frozen at the last freely licensed version) that runs HCL inside Crossplane, wrapping your existing modules as a managed resource. You can reuse your whole investment in Terraform and still gain continuous reconciliation for whatever needs it. If you already have Terraform in production, this hybrid pattern is usually your sensible route, not the rewrite.

On comparative adoption you have to be sober: there is no official, reliable “Crossplane against Terraform” figure, and the most repeated documented criticism is that Crossplane lacks the mature module ecosystem Terraform has been accumulating for years. That is a real qualitative difference, not a detail.

The awkward question: is it an AWS-style CLI?

Since on this blog I have been pulling on the thread of CLI provisioning, I will answer directly: no. Crossplane does not offer you an imperative aws s3 create-bucket style experience. It is declarative first. You provision with a kubectl apply of a resource, and the crossplane binary is there to build, package, test and debug, not to “create a bucket” in one go.

The subcommands make that clear. crossplane render renders a composition and its functions locally, so you can test without a cluster. The crossplane xpkg build, push, install family manages the OCI packages of providers, configurations and functions. And crossplane beta validate, trace and top validate schemas, draw the resource tree and show consumption inside the cluster. It is the toolbox of whoever builds the platform, not of whoever consumes it day to day. Upbound’s up CLI adds more, but oriented towards its commercial product.

The conclusion matters for setting expectations: the imperative AWS-console-style experience, if you want it, gets built on top of Crossplane, with Backstage, a portal or your own wrapper. Crossplane gives you the declarative API; the friendly face you put on separately. If what you are after is an imperative provisioning CLI, you fit better with the model of KubeVela and Score, which is precisely the subject of the next article.

On-premise providers: provisioning without touching AWS

For a sovereign platform your question is what you can provision without setting foot in a public cloud, and the answer is more than it seems, with uneven maturity.

The most versatile is provider-kubernetes, which manages any Kubernetes object in remote clusters through an Object resource. It is the basis of the multi-cluster on-premise pattern: from a central cluster you materialise resources in your secondary RKE2s. Alongside it, provider-helm installs and manages Helm charts declaratively through a Release resource, letting you deploy things like vLLM or KServe as part of a composition.

For the identity and storage part you have concrete pieces: provider-keycloak configures Keycloak realms, clients and roles declaratively, useful for the sovereign IAM we already set up in the article on authentication with Keycloak; and provider-minio manages buckets and policies of on-premise S3-compatible storage. For vSphere you do not have a first-class native provider, but you cover it by generating one from the Terraform provider, the usual pattern for VMware infrastructure.

An honest warning: for KubeVirt, Harvester or CloudNativePG there are no mature, first-class native providers; in practice you manage them by applying their CRDs through provider-kubernetes or by wrapping their Terraform. It works, but it is not the polished experience of a dedicated provider. With the combination of provider-kubernetes, provider-helm, provider-minio and provider-keycloak, plus provider-terraform for whatever is missing, you can assemble a one hundred per cent on-premise self-service catalogue, with the caveat that maturity varies piece by piece.

The fit with GPU inference: from pattern to Modelplane

This is the section that in the first version of this text carried the most caution, because there was no public case of Crossplane orchestrating GPU and inference and it all remained a design pattern. Halfway through 2026 that changed, and the case was published by the very team that makes Crossplane.

Modelplane is an open source control plane for AI inference, Apache-licensed and with a first version 0.1 from June 2026, and the interesting thing is how it is built: entirely on Crossplane compositions and composition functions in Python, with no controllers of its own. It composes vLLM directly, with the vllm/vllm-openai image, without going through KServe, and its self-service API for the ML teams is a ModelDeployment resource where you declare the model, the engine, the topology and the hardware. Underneath, a ModelReplica resource represents each replica, and the platform team declares its GPU pools with InferenceCluster and InferenceClass resources, using an attribute model and CEL expressions taken from Kubernetes dynamic resource allocation. Its most advanced piece is a two-level fleet scheduler written as a pure function, which places replicas across the whole infrastructure and delegates to each cluster’s scheduler. It is proof that the pattern works, with the warning that it is a version 0.1 under open development, not something to put into production blindly, and that adopting it ties your platform to its resource types.

The other recent enabler is in KServe, which introduced a type dedicated to large models, LLMInferenceService, still in alpha and built on the llm-d project, with prefill and decode separation and multi-node execution, and which runs vLLM as its runtime. It is different from the classic InferenceService, reserved for predictive ML. That new type is your ideal brick if you prefer composing KServe instead of bare vLLM.

With that, your own golden path stops being a hypothesis. You define an XRD, for example InferenceEndpoint, with fields such as the model, the number and type of GPU and the replicas. Its composition, taking advantage of the fact that in v2 it can include any Kubernetes resource, materialises in a single pipeline the team’s namespace, the quota for nvidia.com/gpu and a KServe LLMInferenceService pointing at the model with vLLM runtime, and the developer only applies the InferenceEndpoint with four fields, knowing nothing about KServe or the quotas underneath. KServe supports vLLM and GPU natively, as we saw in the article on KServe.

The honest reading is that you are at the frontier, not on well-trodden ground. Outside Modelplane, made by Upbound, there is still no body of independent cases assembling Crossplane with vLLM, and the KServe type is in alpha. It is a coherent architecture, with a real reference backing it and plenty of room for you to be among the first to document it on your own iron.

What it costs you to operate your own control plane

No tool that gives you back control is free, and Crossplane has well-documented costs you need to budget for beforehand, not discover afterwards.

The learning curve is steep, acknowledged even in the coverage of its graduation. Adopters report debugging difficulties when a composition or a provider misbehaves, and they miss Terraform’s module ecosystem.

Compositions tend towards complexity. There are practitioners describing definitions spanning thousands of lines with poor navigation, and the absence of an integrated testing framework. Functions improve this, but as I said, they shift the complexity rather than remove it.

There is no plan. Unlike Terraform’s plan and apply cycle, Crossplane applies changes immediately. You lose the safety net of reviewing what is going to happen before it happens, with the risk of modifications you did not want.

Operating the control plane itself costs. Big providers create hundreds of resource definitions (the GCP one reached 347), with a risk of API server unavailability and of leaving your cluster hanging. State lives in etcd, which complicates disaster recovery and migrating control between clusters. And there is one specific danger: accidental deletion of cloud resources when upgrading XRDs, providers or compositions, because of the ownership Crossplane keeps over what it created.

Some have decided not to adopt it. The Masterpoint team ruled it out for the lack of data sources and ignore_changes, for incomplete coverage of AWS services, and for the cost of abandoning their accumulated Terraform experience. It is a legitimate decision, and quoting it to you is part of an honest article.

How it lives alongside what you already have

If you already run a platform, your question is not Crossplane yes or no, but Crossplane with what. Three relationships matter to you.

With Backstage they are complementary, not rivals. Backstage is the portal and the templates that generate manifests or pull requests, day zero; Crossplane provisions and reconciles the real state, day two. The usual pattern, which some call the golden triangle, is Backstage as the interface, Crossplane as the control plane and Argo CD or Flux as GitOps. Each one does a layer.

With Flux, which you probably already use, the relationship is additive. Flux reconciles manifests towards the clusters, but it does not give you a platform abstraction API: it does not know what “a database” is, it only applies the YAML you hand it. Crossplane adds exactly that abstraction layer, and the two coexist, with Flux deploying the composite resources and Crossplane materialising them. If you already have Flux, Crossplane’s value is precisely that self-service API layer, in exchange for the cost of operating the control plane I described above.

With KubeVela and Score, the subject of the next article, the boundary is one of layer: Crossplane answers “give me a Postgres”, from the infrastructure side; KubeVela and Score answer “deploy my application”, from the workload side. They do not compete, they stack.

For your sovereign inference factory

The practical decision boils down to this. Crossplane makes sense when you want to expose a self-service API of your own over your on-premise infrastructure and you are willing to operate a control plane to get it. If your platform is used by two teams who know each other and you provision with a tidy Flux repository, Crossplane is cost without return for now. The trigger is scale: when manual provisioning turns into a ticket bottleneck, the declarative API starts paying back its cost.

Your sensible entry route is not rewriting everything. It is starting with a provider that already serves you (provider-kubernetes and provider-helm cover a lot), wrapping your existing Terraform with the corresponding provider instead of throwing it away, and modelling a single abstraction of clear value, probably the InferenceEndpoint we discussed, as a golden path. From there, you grow on demand. And always with a clear account of having traded the safety net of plan and a handful of scripts for a thermostat you have to keep switched on.

The next article moves up a layer, from the infrastructure resource to the application workload, with KubeVela and Score, which is where the friendly CLI that Crossplane deliberately does not give you finally appears.

Sources

CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
Crossplane Blog, Announcing Crossplane’s CNCF Graduation — https://blog.crossplane.io/crossplane-cncf-graduation/
Crossplane Blog, Announcing Crossplane 2.0 — https://blog.crossplane.io/announcing-crossplane-2-0/
Crossplane Docs, What’s New in v2 — https://docs.crossplane.io/latest/whats-new/
Crossplane Docs, Compositions — https://docs.crossplane.io/latest/composition/compositions/
Crossplane Docs, Release Cycle — https://docs.crossplane.io/latest/learn/release-cycle/
Crossplane Docs, Crossplane CLI — https://docs.crossplane.io/latest/cli/
Upbound, Crossplane Graduates From CNCF, Upbound Redefines AI-Native Infrastructure — https://www.upbound.io/blog/crossplane-graduates-from-cncf-upbound-redefines-ai-native-infrastructure
InfoQ, Crossplane Reaches Production Maturity by Graduating CNCF — https://www.infoq.com/news/2025/11/crossplane-grad/
Pulumi Docs, Pulumi vs. Crossplane — https://www.pulumi.com/docs/iac/comparisons/crossplane/
Upbound Marketplace, provider-terraform — https://marketplace.upbound.io/providers/upbound/provider-terraform/latest
GitHub, provider-kubernetes — https://github.com/crossplane-contrib/provider-kubernetes
GitHub, provider-helm — https://github.com/crossplane-contrib/provider-helm
GitHub, provider-keycloak — https://github.com/crossplane-contrib/provider-keycloak
VSHN, provider-minio — https://github.com/vshn/provider-minio
Masterpoint, Crossplane: Why it Didn’t Work for Us — https://masterpoint.io/blog/passing-on-crossplane/
CECG, Crossplane: the good, the bad and the ugly — https://www.cecg.io/blog/crossplane-the-good-the-bad-the-ugly
Taloflow, Backstage vs Crossplane for Platform Engineering — https://www.taloflow.ai/guides/comparisons/backstage-vs-crossplane-platform-engineering
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/
Modelplane — https://modelplane.ai · https://github.com/modelplaneai/modelplane
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview

From public cloud to private: what you really lose and which projects fill the gap

Mon, 31 Aug 2026 09:15:00 +0200

With this one I open a run about the biggest gap a sovereign platform leaves you: how to give your teams the self-service experience of a public cloud, without the public cloud. Here you have the map of the problem; the next two go down into the detail of the tools, Crossplane as the infrastructure control plane and KubeVela with Score as application-centric self-service.

TL;DR

Leaving the public cloud has a solid business case, and I am not making it up. 37signals declared savings of close to two million dollars in 2024 after repatriating, the classic a16z analysis puts cloud spend at around half of the cost of revenue of a software company, and the Barclays CIO survey of 2024 recorded the highest repatriation intent in its history. If your workload is a predictable, sustained base (which is exactly the continuous LLM inference this blog is about) the numbers work out, and in Europe the sovereignty reasons pile on top: the Data Act, the tension with the US CLOUD Act, and that dependence on three hyperscalers that control around 70 % of the European cloud market.

But the problem is not the compute. Your on-premise GPU rack serves tokens just as well as a rented one. What the cloud was selling you and you now have to rebuild is something else: the elasticity of asking for a hundred machines at ten and handing them back at eleven, the catalogue of hundreds of managed services with an on-call team behind them, the unified provisioning API with fine-grained permissions, the multi-region network, the bill that arrives itemised, and that model where operating all of it was somebody else’s problem. All of that you rebuild piece by piece with open source projects, and the rebuild has a cost that the savings figures do not usually tell you about. What follows is the map of what you lose, which project fills each gap, and where a gap remains that nothing will close completely.

The analogy: leaving the hotel to set up your own house

Think of it like this: living in the public cloud is living in a hotel. You pay a lot per night, but you do not change light bulbs, reception is open at three in the morning, if you need another room you ask for it and it appears, and when you leave you take nothing with you to maintain. Setting up your own house works out much cheaper per square metre if you are going to stay for years, and on top of that you decide who comes in and where your things are kept. The problem arrives the day a pipe bursts: then you do not call reception, because reception is you.

Repatriation is that move. The savings are real and so is data ownership, but what in the hotel came included in the price of the night is now your to-do list: the plumbing is your network, the boiler is your storage, the concierge who got you anything is the catalogue of managed services you no longer have, and the itemised bill they slid under your door is a cost model you have to build, because electricity and depreciation do not arrive broken down per tenant. The house pays off, but only if you count what it costs to furnish it and fix the pipes, not just the price per square metre against the hotel.

The rest of the article walks through that to-do list and tells you, for each item, which open source tool does the job the hotel used to do for you.

Why this conversation belongs to 2026, not to always

Repatriating is nothing new, but in 2026 three things come together that make it urgent if you run a European AI platform.

The first is economic, and Andreessen Horowitz put it on the table in 2021 with The Cost of Cloud, a Trillion Dollar Paradox. Their thesis, which you should read knowing that a16z has interests in the ecosystem it analyses, is that cloud spend averages around 50 % of the cost of revenue in a software company, and that recovering half of that spend would free up margins the market penalises. The cleanest case, and one you can verify, is 37signals: DHH published that their cloud bill dropped from 3.2 to 1.3 million dollars a year after repatriating, with an initial investment of about 700,000 dollars in Dell servers. Now the honest caveat, which the specialist press itself underlines: those savings figures did not include the extra operations staff, nor the energy, nor the cooling. I come back to it in the hidden costs section.

The second is declared intent. The Barclays CIO survey from the first half of 2024 recorded that 83 % planned to repatriate some workload to private cloud or on-premise in the following twelve months, the highest reading in the series. But read that 83 % precisely: it is the proportion of managers who plan to move some workload, not the percentage of workloads or of spend that moves. Flexera, in its 2025 report, gives you the more grounded figure: large companies have repatriated around 21 % of the workloads they had in the public cloud, and cost saving is priority number one for almost 60 %.

The third is regulatory and European, and it is the one that turns repatriation into something more than a FinOps decision. The European Union Data Act came into force in January 2024 and has applied since September 2025; it mandates portability between providers and, from January 2027, bans the exit fees that today chain you to your provider. And underneath it beats a deeper legal tension: the US CLOUD Act allows United States authorities to demand data from an American provider even if it is stored in Frankfurt, which is exactly the argument for why a European region of a hyperscaler does not give you sovereignty. With AWS, Microsoft and Google controlling around 70 % of the European cloud market, the conversation about sovereign cloud stopped being theoretical. I developed this already in sovereign on-premise versus hyperscalers.

What you lose when you leave the cloud, specifically

Let us get to the core, ordered by what is going to hurt you most. And one thing before starting: none of these gaps is about compute. They are all about what the cloud put around the compute.

Hardware elasticity disappears. On-premise, the GPU you did not buy does not exist. The cloud bills you by the hour precisely because it takes on the risk of idle capacity: it lets you ask for a peak and hand it back. With your own iron, you buy the peak up front and pay for it whether it is in use or not, so your profitability depends on high sustained utilisation. Below a certain usage threshold, the cloud beats you again. This is the irreducible gap, the one no tool fills, and if you repatriate a workload with extreme peaks you are making the wrong decision.

Managed services become your operations. Every box you took for granted (the database, the queue, the object storage, the load balancer, DNS, the secrets manager) stops being a checkbox you tick and becomes a system you install, update and watch yourself. The 37signals case teaches it well: getting out of S3 object storage was the last and hardest part, with petabytes of data and a contract they dragged along for years. Your managed Postgres turns into operating CloudNativePG; your S3 turns into operating MinIO; and so on with every service.

You have to build the provisioning control plane yourself. AWS gives you a unified API where you ask for a resource with fine-grained permissions and it appears. On-premise that does not come for free: you have to build the control plane that translates “I want a database” into real resources. It is exactly the gap that Crossplane fills, and that is why it has its own article.

You replace the managed network piece by piece. The VPC, the security groups, NAT, the load balancer: every network abstraction in the cloud has its on-premise equivalent (Cilium for network policies on eBPF, MetalLB for the load balancer without ELB, Gateway API for L7 ingress), but they are pieces you assemble, not a service you switch on. A good part of this we already saw in the networking vertical with Cilium and eBPF.

Billing stops existing as such. In the cloud the cost reaches you broken down by service and by tag; you have a Cost Explorer that answers “how much does this team cost me”. On-premise there is no natural bill: there is hardware depreciation, an electricity bill and some payroll, and splitting that per workload is a model you have to build. OpenCost helps you with the split per workload inside the cluster, but the total physical cost (depreciation plus energy plus staff) you model separately, as we saw in OpenCost and cost allocation.

You design multi-zone availability yourself. The three availability zones the cloud gives you for free turn into several sites, several racks and several utility feeds that you have to plan. 37signals solved it with two datacenters and replication between them. It is neither free nor automatic.

And above all, the on-call is yours. The cloud was not selling you only compute: it was selling you operations included. When something falls over at three in the morning, in the cloud there is an on-call team at the provider; with open source, that team is you. This is the cost that savings spreadsheets almost never include, and it is the one that decides whether your repatriation was a good idea or a hidden bill.

The map of projects that fill each gap

With the gaps ordered, the catalogue of substitutes reads itself. I group it by layer, from the lowest to the closest to the developer.

In the infrastructure substrate, what replicates EC2 virtual machines, virtual networking and disks, you have OpenStack as the classic standard for a complete private cloud, SUSE’s Harvester as bare-metal hyperconvergence on KubeVirt managed from Kubernetes, and emerging projects such as Spinifex that rebuild an AWS-compatible API on your own iron. It is the heaviest layer and the one I touch least in this blog, because I assume your cluster already exists.

In the provisioning control plane, what replicates the AWS IAM and resource API, there is Crossplane, graduated in the CNCF in November 2025, which brings the Kubernetes declarative API model to infrastructure. Alongside it, Cluster API for declarative cluster provisioning. And so you can see how far this reaches in our own field, the Crossplane team itself published Modelplane in mid-2026, an inference control plane built on compositions that serves models with vLLM. This layer is developed by the Crossplane article.

In the developer self-service experience, what replicates the AWS console and CLI, you have three different approaches you should not confuse. Backstage is the web portal, the catalogue and the templates, and it already has its own article. KubeVela brings self-service to the application model with a friendly CLI. Score defines a workload specification that is portable between environments. KubeVela and Score share the third article because they cover the same layer from angles that complement each other.

In on-premise managed services, CloudNativePG operates Postgres for you with high availability and backups to object storage, MinIO gives you S3-compatible storage, and OpenCost covers that cost allocation you lost with Cost Explorer.

The way to read this map is that none of these pieces is surplus if you want the cloud experience: they are the boxes the cloud ticked for you and that you now tick one by one.

The gap you will not close

I will tell you plainly, because it is the part the “leave the cloud” brochures skip. No combination of open source projects reproduces 100 % of the public cloud for you, and not for lack of maturity, but by design.

What you will not replicate is, first, instant hardware elasticity: you still buy the peaks up front. Second, the depth and breadth of the managed services catalogue, hundreds of services with a service level agreement and an on-call team behind them, which no internal team matches. Third, the global multi-region network the cloud gives you almost for free. And fourth, that model of transferred operational responsibility: with open source, operations come back home.

The correct reading is not that repatriating is a bad idea, but that you swap a predictable operating expense and a delegated responsibility for a cheaper capital investment and a responsibility of your own. For sustained base load it works out. For everything else, think about it.

When NOT to repatriate

This section has to be here, because repatriation badly applied is expensive. There are three clear signs that the cloud is still your right answer.

The first is a workload with extreme peaks or very variable traffic. Most inference deployments operate between 40 and 65 % GPU utilisation, and below roughly 70 % sustained usage the cloud beats you on total cost. If your workload has long troughs, you are buying iron so it can sleep.

The second is uncertainty or the start of a project. The break-even point in favour of buying only applies to you when utilisation is consistently high and, above all, measured and not projected. Buying GPUs against an optimistic forecast is the fastest way to end up with a warehouse of idle silicon. Sizing it properly is worth a whole article: sizing and justifying the GPU investment.

The third is the lack of an operations team. The total cost of ownership on-premise includes between half and one full-time engineer per GPU server. Without that team, your saving is illusory, because the on-call the cloud included is now covered by nobody. a16z itself frames its thesis as a hybrid approach, not as “leave the cloud” at all.

The numbers, to ground it

So as not to leave you with the discussion in the abstract, here are the orders of magnitude, triangulating two sources with opposite biases, a cloud provider and a neutral FinOps tool. An H100 in the cloud runs you between 2.90 and almost 7 dollars per GPU-hour depending on provider; bought, an H100 card is around 31,000 dollars and an eight-way system between 250,000 and 320,000. At full utilisation, a rented H100 passes its purchase price in less than a year, and an eight-way system with second-hand hardware can pay for itself in about seven months. The three-year total cost of an eight-H100 system, counting staff, colocation, energy and cooling, is estimated at between 700,000 and 950,000 dollars, and the threshold where on-premise beats the hyperscalers is around 80 % sustained utilisation.

The two sources disagree on the exact number, as you would expect, but they agree on the deciding factor: sustained utilisation, measured and not projected. The full working of this calculation is in the TCO of an on-premise GPU cluster. Here the conclusion is enough for you: for continuous base load the numbers work out, and that is exactly the profile of an inference factory.

For your sovereign inference factory

Everything above arranges itself into a practical decision. If you operate an LLM inference platform on your own iron, you are not deciding whether to repatriate (you already took that decision when you built the cluster), but how to rebuild the cloud experience your teams miss.

The sensible sequence starts with the substrate, which you probably already have in the form of RKE2 and storage. It goes on to the provisioning control plane, where Crossplane turns “open a ticket and wait” into a declarative API. It continues with the developer experience, where you choose between the Backstage portal, the KubeVela application model or the portable Score specification, depending on whether your people prefer a web or a CLI. And do not forget the on-premise managed services or the cost allocation, because without a bill your platform loses the discipline the cloud imposed by default.

The next two articles go down into the two layers least covered in this blog: Crossplane for the infrastructure control plane, and KubeVela with Score for CLI self-service centred on the application. With them, the map of this layer is covered, and your own house starts to look like the hotel you left, with the difference that the data stays inside.

Sources

Andreessen Horowitz (S. Wang, M. Casado), The Cost of Cloud, a Trillion Dollar Paradox — https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/
Data Center Dynamics, 37signals claims it saved almost 2M USD last year from cloud repatriation — https://www.datacenterdynamics.com/en/news/37signals-claims-it-saved-almost-2m-last-year-from-cloud-repatriation/
The Register, Developer pockets 2M USD in savings from going cloud-free — https://www.theregister.com/2024/10/21/37signals_aws_savings/
Barclays, Technology: 1H24 CIO Survey (PDF) — https://8198920.fs1.hubspotusercontent-na1.net/hubfs/8198920/Barclays_Cio_Survey_2024-1.pdf
Channelnomics, Breaking Down the 83% Public Cloud Repatriation Number — https://channelnomics.com/breaking-down-the-83-public-cloud-repatriation-number/
The New Stack, Updated Stats on Cloud Sustainability, Repatriation and Cost Optimization (Flexera 2025) — https://thenewstack.io/updated-stats-on-cloud-sustainability-repatriation-and-cost-optimization/
Comisión Europea, Data Act | Shaping Europe’s digital future — https://digital-strategy.ec.europa.eu/en/policies/data-act
EU Data Act, Article 29 — Gradual withdrawal of switching charges — https://www.eu-data-act.com/Data_Act_Article_29.html
Kiteworks, How the EU Data Act and GDPR Conflict with U.S. CLOUD Act — https://www.kiteworks.com/gdpr-compliance/eu-data-act-gdpr-cloud-conflict/
Computerworld, EU takes first steps to reduce reliance on US hyperscalers — https://www.computerworld.com/article/4181816/eu-takes-first-steps-to-reduce-reliance-on-us-hyperscalers.html
CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Harvester HCI (SUSE) — https://harvesterhci.io/
CloudNativePG — https://cloudnative-pg.io/
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
Spheron, LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
CloudZero, H100 GPU Cost in 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/

Backstage as a self-service portal: giving your teams the LLM platform without giving them the cluster

Mon, 31 Aug 2026 09:00:00 +0200

This article opens the platform and self-service layer: the one that decides how you hand everything underneath to the teams that are going to use it. The three articles in this batch are Backstage (the portal), Kubeflow (the ML toolbox) and Knative (switching the GPU off when nobody is using it).

TL;DR

Backstage solves a real problem of a platform that has grown: nobody knows what is there, who maintains it, or how to ask for a new one without opening a ticket. Its answer is three pieces, a software catalogue that models every component and its relations, templates that create new resources through a form instead of through copy and paste, and TechDocs to keep documentation next to the code. On top of those three sits the concept that holds up everything else, the golden path: the supported route made also the easiest one, so that people take it because it is paved, not because they are forced onto it.

The price is high and it should be said plainly. Backstage is not a product you install, it is a Node and React application that you compile and maintain as your own code, at a declared cost of two to three full-time engineers, and with upgrades as its biggest pain point. As of August 2026 it is still a CNCF incubating project, not graduated, and it has no native entity type for models, datasets or inference endpoints: that gets modelled by hand, or you wait for a proposal that is still open. The decision is not about features, it is about scale: below a certain organisation size, a portal costs more than it saves.

The analogy: the industrial estate directory

A single unit needs no sign. You know what is inside because you built it. An estate with two hundred units, on the other hand, needs a directory at the entrance: which company sits on each plot, what it does, who is responsible, and an office where you can request a new plot with the utility connections already in place instead of negotiating each one separately.

An LLM platform starts out as a single unit. One cluster, one model, the team that built it. Six months later it is an industrial estate: fifteen models served, three RAG pipelines, LoRA adapters everywhere, and a question that comes back in every meeting, who maintains this? Backstage is the directory for that estate, with two additions an ordinary directory does not have: the office that hands over turnkey plots, and the archive of blueprints for each unit. The catalogue is the directory, the templates are the office, TechDocs is the archive.

The analogy also warns about the cost. The directory does not maintain itself. Somebody has to update it when a company moves, and a directory with half the plots mislabelled is worse than no directory at all, because people stop trusting it. In Backstage that failure has a name, catalog rot, and it is the leading cause of a portal being abandoned.

What Backstage is and what state it reaches 2026 in

Backstage was born inside Spotify and donated to the CNCF, which accepted it in September 2020 and moved it to incubating level in March 2022. That is where it still is: as of August 2026 it is incubating, not graduated, a detail that matters when someone sells it as a de facto standard. It is by far the most widely adopted developer portal (the official 2025 review declares more than 3,400 organisations using it, 31,000 GitHub stars and more than 250 plugins), but adoption is not governance maturity, and a large share of those plugins is maintained by the community, not by the core.

The architecture has kept changing pieces under the feet of whoever operates it. The New Backend System stabilised in 2024 and is now the standard. The New Frontend System became the default mode for new applications in version 1.49, of March 2026. The cadence is one minor release a month, and that rhythm is precisely the origin of the biggest operational problem, which we will see further down.

The three pieces

The software catalogue

The heart of Backstage is an entity catalogue. Every piece of the platform is described in a catalog-info.yaml file that lives next to its code, and Backstage discovers it by crawling the repository. The entity types are fixed: Component (something that runs or compiles), API (a contract that a component exposes), Resource (infrastructure that a component needs), System (a set of components that form a unit), Domain (a set of systems), plus Group and User for ownership.

The power is in the relations rather than in the entities, which Backstage computes and navigates: ownedBy connects each thing to its team, providesApi and consumesApi draw who depends on whom, partOf groups components into systems, dependsOn traces the dependency graph. With that, the meeting question (who maintains this, and what breaks if we touch it?) stops being answered from memory and starts being answered by query.

There is a design decision that conditions everything that follows: Backstage advises against creating new entity types. Plugins are compiled expecting the usual types, and an invented type leaves them blind. The supported route is to reuse an existing type and distinguish it with the spec.type field (for example a Resource with type: llm-model), adding whatever custom fields you need to the spec.

The templates

The second piece is the Scaffolder, the template engine. A template is an entity of type Template with three parts: the parameters requested through a form, the steps that get executed, and the actions that do the real work (create a repository, write the catalog-info.yaml, register the component, trigger the GitOps pipeline). The developer goes to /create, picks a template, fills in the form, and in a single operation has the resource created, catalogued and deploying.

This is what turns a directory into an office. Without templates, Backstage only lists what already exists. With templates, it is the door through which new things are created, and that door is where the golden path materialises.

TechDocs

The third piece is TechDocs, documentation as code. Documents are written in Markdown next to the code, generated with MkDocs in the CI pipeline, and served inside the portal itself, linked from each component’s page. The advantage is in the incentives: documentation that lives far from the code rots; documentation that lives next to it and gets reviewed in the same pull request has some chance of staying true.

The golden path, which is what all of this is about

Spotify coined the term golden path for the recommended and supported way to build something. It has two halves: the blessed tooling, visible in the portal, and a step-by-step tutorial in the documentation. The idea, in the formulation that platform reports keep repeating, is that nobody forces you to use the paved road; people take it because it is paved.

The data supports the golden path being the deciding variable. The 2026 State of Platform Engineering report, covering 518 professionals, puts voluntary adoption above 80 % where there is a well-designed golden path and below 20 % where there is not, and estimates that around 70 % of platform initiatives fail to reach significant adoption without one. It is a figure that reorders priorities: a portal without golden paths is a pretty catalogue nobody uses.

In Backstage, a golden path is implemented as a Scaffolder template. For an LLM platform, a typical golden path would be “serve a new model”: the form asks for the model, the GPU size, the owning team and the SLO; the template creates the repository with the InferenceService, writes the catalog-info.yaml, registers it under the right system, and opens the pull request against the GitOps repository you already have with Flux. What used to be half a day of copying a colleague’s YAML and guessing the values becomes a two-minute form with the correct defaults.

Authentication and permissions

For a sovereign platform the identity piece matters as much as the catalogue. Backstage ships connectors out of the box for GitHub, GitLab, Google, Microsoft, Okta, generic OIDC and SAML, among others. For on-premise the relevant one is Keycloak, which fits in like any other OIDC provider, with one nuance worth keeping in mind: the Keycloak connector is maintained by the community, not by the core, so its maintenance and its tightness fall on a third-party plugin. Once you have set up Keycloak as we saw in the article on authenticating MCP, the portal hooks into the same identity issuer.

The permissions framework appears today as stable in the documentation and supports RBAC, ABAC and custom logic. The small print: by default the endpoints are not protected, and the permission policy is written by the integrator. The most widely used RBAC solution with a graphical interface comes with the Red Hat distribution, not with the core. In other words, serious access control means work on top of what comes in the box.

The real cost, no frills

This is where an honest article parts ways with the brochure. Backstage is not an installable product, it is a framework that you compile and maintain as your own code. That has measurable consequences.

The most quoted analysis, from the managed Backstage vendor Roadie, puts the effort at around three full-time engineers in the first year and two from the second onwards, with a time to production of six to twelve months in self-managed mode. The figure comes from a vendor with a commercial interest in you buying its managed version, so it should be taken as an order of magnitude and not as a neutral data point, but the direction is confirmed by independent sources: ongoing maintenance takes up around a fifth of engineering time, and upgrades are the biggest pain point, cited by more than half of those who adopt it. With one minor release a month and architectural pieces shifting under your feet, keeping up to date is a permanent job.

The other cost is about adoption, not engineering. A widely repeated critical analysis points out that many organisations settle at around 10 % internal usage, against the 99 % voluntary usage Spotify achieved at home, and attributes it to three recurring failures: the catalog rot we already mentioned, the lack of a product owner treating the portal as a product, and the illusion that Backstage can be adopted over what already exists when in practice it only standardises new things well. The sentence that sums up the disenchantment is that Backstage is a free portal in the same way you would have a free car if somebody tipped all the parts of a Chevrolet onto your desk.

Applied to an LLM platform: what is there and what is not

Here an expectation needs correcting. In 2026 Backstage has no native entities for models, datasets or inference endpoints. There are three things and none of them is what you would hope for:

An open proposal (RFC 33060, from February 2026) to model model servers as entities of type API with type: ai-model-server. It is under discussion, not implemented.

An official module that introduces an AiResource type, which sounds like what we are after but is for something else: governing the skills and rules of coding assistants, not registering inference models.

And one real pattern, the MLOps templates that AWS documents on top of SageMaker, where Backstage acts as the Scaffolder that creates environments and pipelines, not as the model registry. It is a cloud case, not on-premise, and it uses Backstage for its template half, not for its model catalogue.

The practical conclusion for your own stack is twofold. First, what Backstage brings to an LLM platform today is above all the template half: the “create a new inference service” golden path wired into your GitOps. Second, if you want to catalogue models, today that is done by hand with custom spec.type values, treating the model as a Resource and its endpoint as an API, until the proposal matures. Registering the model’s bytes remains the job of the OCI registry, not of the catalogue.

The infrastructure plugins that are worth it in this stack do exist and are reasonably alive: the Kubernetes one (core), the Flux one for seeing GitOps state from the portal, the Grafana one for embedding dashboards in a component’s page, and the ArgoCD one for anyone using Argo instead of Flux.

Alternatives: when not to set up Backstage

The honest decision is not which portal, but whether a portal. The SaaS-type alternatives (Port, Cortex, OpsLevel, Humanitec) win on time to first value, measured in days instead of months, and on not having to maintain the framework, in exchange for a per-user fee that scales badly and for giving up pure on-premise. Roadie offers managed Backstage, which removes the framework maintenance while keeping its extensibility, in exchange for a per-developer fee and for taking your catalogue outside.

For a sovereign platform, where on-premise and data ownership are requirements, pure SaaS is usually ruled out by design, which leaves the real choice between self-managed Backstage and no portal at all. And there the criterion is size. Below a handful of teams, a well-maintained README and a tidy GitOps repository do the same job as the catalogue, without the cost of two engineers. The portal starts paying off when the cognitive load of knowing what is there and how to ask for it exceeds the cost of maintaining the tool that answers it.

Operational traps and honest scepticism

The catalogue rots if it is not automated. A catalog-info.yaml written by hand and never updated is lying within a few weeks. The only defence is generating and validating those files in the pipeline, not trusting people to maintain them.

Without a product owner, the portal dies. Backstage is not infrastructure you install and forget, it is an internal product with users you have to understand. Without somebody treating adoption as an objective, it ends up as a catalogue nobody looks at.

Upgrades are a recurring tax. One release a month and architectural pieces mid-migration mean the portal is never finished. That maintenance has to be budgeted from the start, not discovered in the second quarter.

Backstage deploys nothing. It is a portal, not an engine. Behind each template there has to be your GitOps, your operators and your cluster doing the real work. If that base is not solid, the portal just puts a pretty façade over chaos.

The data model for AI does not exist yet. Anyone setting up Backstage expecting a ready-to-use model catalogue is going to find themselves modelling by hand. It is doable, but it is work, and you should know that beforehand.

For an inference factory

Backstage fits at the high maturity level of a platform, not at the start. In the five maturity levels it is a tool of the level where the platform already serves several teams and the bottleneck has stopped being technical and become one of coordination. Three concrete decisions follow from all of the above.

The first is not setting it up too early. If the platform is used by two teams who know each other, the portal is cost without return. The trigger is cognitive load, not the calendar.

The second is starting with the templates, not with the catalogue. The value you notice from day one is the golden path that turns half a day of copying YAML into a two-minute form. The complete catalogue, with all its relations, is a bigger job that can come later.

The third is budgeting maintenance as a function, not as a project. Two part-time engineers indefinitely, an owner watching adoption, and automatic generation of the catalogue entries. Without that, the portal degrades until somebody asks why nobody uses it, and the answer is that it stopped being true.

The portal is the face your platform shows to whoever uses it. It is worth it when there is enough platform behind it that showing it tidily saves more than maintaining the shop window costs. Before that, it is the shop window of a shop that has no stock yet. The other two articles in this batch go into that stock: the ML toolbox with Kubeflow and the savings of switching the GPU off with Knative.

Sources

CNCF, Backstage (project page) — https://www.cncf.io/projects/backstage/
Backstage, Backstage Wrapped 2025 — https://backstage.io/blog/2025/12/30/backstage-wrapped-2025/
Backstage Docs, Software Catalog: descriptor format — https://backstage.io/docs/features/software-catalog/descriptor-format/
Backstage Docs, Well-known relations — https://backstage.io/docs/features/software-catalog/well-known-relations/
Backstage Docs, Extending the model — https://backstage.io/docs/features/software-catalog/extending-the-model/
Backstage Docs, Software Templates — https://backstage.io/docs/features/software-templates/
Backstage Docs, TechDocs — https://backstage.io/docs/features/techdocs/
Backstage Docs, Authentication — https://backstage.io/docs/auth/
Backstage Docs, Permissions overview — https://backstage.io/docs/permissions/overview/
Backstage GitHub, RFC #33060: modeling AI model servers in the catalog — https://github.com/backstage/backstage/issues/33060
Roadie, Backstage: how much does it really cost? — https://roadie.io/blog/backstage-how-much-does-it-really-cost/
Earthly, Backstage is at the peak of its hype — https://earthly.dev/blog/backstage-is-at-peak-hype/
AWS Prescriptive Guidance, Accelerate MLOps with Backstage and SageMaker templates — https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-mlops-with-backstage-and-sagemaker-templates.html
Spotify Engineering, How we use golden paths to solve fragmentation — https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem
The New Stack, DORA 2024: AI and platform engineering fall short — https://thenewstack.io/dora-2024-ai-and-platform-engineering-fall-short/
bex.co, State of Platform Engineering Report Vol. 4: golden path — https://bex.co/blog/2026/08/06/platform-engineering-report-vol-4-golden-path

Autoservicio on lo0 — Tech Blog

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

TL;DR

The analogy: the order form against the shopping list

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

The OAM model in five pieces

CUE, the power and the toll

The vela CLI: Heroku on the outside, GitOps on the inside

Multi-cluster, which is what matters to you

VelaUX, the lightweight portal thrown in

Score: describe the workload once

What it is and what it solves

The two implementations

Score against KubeVela and Helm

Where each one fits: the cross-cutting comparison

The fit with inference: a pattern, not an official recipe

Caveats, no frills

For your sovereign inference factory

See also

Sources

Crossplane: the control plane that turns your cluster into a private cloud API

TL;DR

The analogy: the thermostat against the light switch

What Crossplane is and what state it reaches 2026 in

The v2 turn (which changes how you think about it)

The model, piece by piece

Composition functions: the real logic

Crossplane against Terraform, OpenTofu and Pulumi

The awkward question: is it an AWS-style CLI?

On-premise providers: provisioning without touching AWS

The fit with GPU inference: from pattern to Modelplane

What it costs you to operate your own control plane

How it lives alongside what you already have

For your sovereign inference factory

See also

Sources

From public cloud to private: what you really lose and which projects fill the gap

TL;DR

The analogy: leaving the hotel to set up your own house

Why this conversation belongs to 2026, not to always

What you lose when you leave the cloud, specifically

The map of projects that fill each gap

The gap you will not close

When NOT to repatriate

The numbers, to ground it

For your sovereign inference factory

See also

Sources

Backstage as a self-service portal: giving your teams the LLM platform without giving them the cluster

TL;DR

The analogy: the industrial estate directory

What Backstage is and what state it reaches 2026 in

The three pieces

The software catalogue

The templates

TechDocs

The golden path, which is what all of this is about

Authentication and permissions

The real cost, no frills

Applied to an LLM platform: what is there and what is not

Alternatives: when not to set up Backstage

Operational traps and honest scepticism

For an inference factory

See also

Sources