From public cloud to private: what you really lose and which projects fill the gap

· By José Manuel Román
Contents

With this one I open a run about the biggest gap a sovereign platform leaves you: how to give your teams the self-service experience of a public cloud, without the public cloud. Here you have the map of the problem; the next two go down into the detail of the tools, Crossplane as the infrastructure control plane and KubeVela with Score as application-centric self-service.

TL;DR

Leaving the public cloud has a solid business case, and I am not making it up. 37signals declared savings of close to two million dollars in 2024 after repatriating, the classic a16z analysis puts cloud spend at around half of the cost of revenue of a software company, and the Barclays CIO survey of 2024 recorded the highest repatriation intent in its history. If your workload is a predictable, sustained base (which is exactly the continuous LLM inference this blog is about) the numbers work out, and in Europe the sovereignty reasons pile on top: the Data Act, the tension with the US CLOUD Act, and that dependence on three hyperscalers that control around 70 % of the European cloud market.

But the problem is not the compute. Your on-premise GPU rack serves tokens just as well as a rented one. What the cloud was selling you and you now have to rebuild is something else: the elasticity of asking for a hundred machines at ten and handing them back at eleven, the catalogue of hundreds of managed services with an on-call team behind them, the unified provisioning API with fine-grained permissions, the multi-region network, the bill that arrives itemised, and that model where operating all of it was somebody else’s problem. All of that you rebuild piece by piece with open source projects, and the rebuild has a cost that the savings figures do not usually tell you about. What follows is the map of what you lose, which project fills each gap, and where a gap remains that nothing will close completely.

The analogy: leaving the hotel to set up your own house

Think of it like this: living in the public cloud is living in a hotel. You pay a lot per night, but you do not change light bulbs, reception is open at three in the morning, if you need another room you ask for it and it appears, and when you leave you take nothing with you to maintain. Setting up your own house works out much cheaper per square metre if you are going to stay for years, and on top of that you decide who comes in and where your things are kept. The problem arrives the day a pipe bursts: then you do not call reception, because reception is you.

Repatriation is that move. The savings are real and so is data ownership, but what in the hotel came included in the price of the night is now your to-do list: the plumbing is your network, the boiler is your storage, the concierge who got you anything is the catalogue of managed services you no longer have, and the itemised bill they slid under your door is a cost model you have to build, because electricity and depreciation do not arrive broken down per tenant. The house pays off, but only if you count what it costs to furnish it and fix the pipes, not just the price per square metre against the hotel.

The rest of the article walks through that to-do list and tells you, for each item, which open source tool does the job the hotel used to do for you.

Why this conversation belongs to 2026, not to always

Repatriating is nothing new, but in 2026 three things come together that make it urgent if you run a European AI platform.

The first is economic, and Andreessen Horowitz put it on the table in 2021 with The Cost of Cloud, a Trillion Dollar Paradox. Their thesis, which you should read knowing that a16z has interests in the ecosystem it analyses, is that cloud spend averages around 50 % of the cost of revenue in a software company, and that recovering half of that spend would free up margins the market penalises. The cleanest case, and one you can verify, is 37signals: DHH published that their cloud bill dropped from 3.2 to 1.3 million dollars a year after repatriating, with an initial investment of about 700,000 dollars in Dell servers. Now the honest caveat, which the specialist press itself underlines: those savings figures did not include the extra operations staff, nor the energy, nor the cooling. I come back to it in the hidden costs section.

The second is declared intent. The Barclays CIO survey from the first half of 2024 recorded that 83 % planned to repatriate some workload to private cloud or on-premise in the following twelve months, the highest reading in the series. But read that 83 % precisely: it is the proportion of managers who plan to move some workload, not the percentage of workloads or of spend that moves. Flexera, in its 2025 report, gives you the more grounded figure: large companies have repatriated around 21 % of the workloads they had in the public cloud, and cost saving is priority number one for almost 60 %.

The third is regulatory and European, and it is the one that turns repatriation into something more than a FinOps decision. The European Union Data Act came into force in January 2024 and has applied since September 2025; it mandates portability between providers and, from January 2027, bans the exit fees that today chain you to your provider. And underneath it beats a deeper legal tension: the US CLOUD Act allows United States authorities to demand data from an American provider even if it is stored in Frankfurt, which is exactly the argument for why a European region of a hyperscaler does not give you sovereignty. With AWS, Microsoft and Google controlling around 70 % of the European cloud market, the conversation about sovereign cloud stopped being theoretical. I developed this already in sovereign on-premise versus hyperscalers.

What you lose when you leave the cloud, specifically

Let us get to the core, ordered by what is going to hurt you most. And one thing before starting: none of these gaps is about compute. They are all about what the cloud put around the compute.

Hardware elasticity disappears. On-premise, the GPU you did not buy does not exist. The cloud bills you by the hour precisely because it takes on the risk of idle capacity: it lets you ask for a peak and hand it back. With your own iron, you buy the peak up front and pay for it whether it is in use or not, so your profitability depends on high sustained utilisation. Below a certain usage threshold, the cloud beats you again. This is the irreducible gap, the one no tool fills, and if you repatriate a workload with extreme peaks you are making the wrong decision.

Managed services become your operations. Every box you took for granted (the database, the queue, the object storage, the load balancer, DNS, the secrets manager) stops being a checkbox you tick and becomes a system you install, update and watch yourself. The 37signals case teaches it well: getting out of S3 object storage was the last and hardest part, with petabytes of data and a contract they dragged along for years. Your managed Postgres turns into operating CloudNativePG; your S3 turns into operating MinIO; and so on with every service.

You have to build the provisioning control plane yourself. AWS gives you a unified API where you ask for a resource with fine-grained permissions and it appears. On-premise that does not come for free: you have to build the control plane that translates “I want a database” into real resources. It is exactly the gap that Crossplane fills, and that is why it has its own article.

You replace the managed network piece by piece. The VPC, the security groups, NAT, the load balancer: every network abstraction in the cloud has its on-premise equivalent (Cilium for network policies on eBPF, MetalLB for the load balancer without ELB, Gateway API for L7 ingress), but they are pieces you assemble, not a service you switch on. A good part of this we already saw in the networking vertical with Cilium and eBPF.

Billing stops existing as such. In the cloud the cost reaches you broken down by service and by tag; you have a Cost Explorer that answers “how much does this team cost me”. On-premise there is no natural bill: there is hardware depreciation, an electricity bill and some payroll, and splitting that per workload is a model you have to build. OpenCost helps you with the split per workload inside the cluster, but the total physical cost (depreciation plus energy plus staff) you model separately, as we saw in OpenCost and cost allocation.

You design multi-zone availability yourself. The three availability zones the cloud gives you for free turn into several sites, several racks and several utility feeds that you have to plan. 37signals solved it with two datacenters and replication between them. It is neither free nor automatic.

And above all, the on-call is yours. The cloud was not selling you only compute: it was selling you operations included. When something falls over at three in the morning, in the cloud there is an on-call team at the provider; with open source, that team is you. This is the cost that savings spreadsheets almost never include, and it is the one that decides whether your repatriation was a good idea or a hidden bill.

The map of projects that fill each gap

With the gaps ordered, the catalogue of substitutes reads itself. I group it by layer, from the lowest to the closest to the developer.

In the infrastructure substrate, what replicates EC2 virtual machines, virtual networking and disks, you have OpenStack as the classic standard for a complete private cloud, SUSE’s Harvester as bare-metal hyperconvergence on KubeVirt managed from Kubernetes, and emerging projects such as Spinifex that rebuild an AWS-compatible API on your own iron. It is the heaviest layer and the one I touch least in this blog, because I assume your cluster already exists.

In the provisioning control plane, what replicates the AWS IAM and resource API, there is Crossplane, graduated in the CNCF in November 2025, which brings the Kubernetes declarative API model to infrastructure. Alongside it, Cluster API for declarative cluster provisioning. And so you can see how far this reaches in our own field, the Crossplane team itself published Modelplane in mid-2026, an inference control plane built on compositions that serves models with vLLM. This layer is developed by the Crossplane article.

In the developer self-service experience, what replicates the AWS console and CLI, you have three different approaches you should not confuse. Backstage is the web portal, the catalogue and the templates, and it already has its own article. KubeVela brings self-service to the application model with a friendly CLI. Score defines a workload specification that is portable between environments. KubeVela and Score share the third article because they cover the same layer from angles that complement each other.

In on-premise managed services, CloudNativePG operates Postgres for you with high availability and backups to object storage, MinIO gives you S3-compatible storage, and OpenCost covers that cost allocation you lost with Cost Explorer.

The way to read this map is that none of these pieces is surplus if you want the cloud experience: they are the boxes the cloud ticked for you and that you now tick one by one.

The gap you will not close

I will tell you plainly, because it is the part the “leave the cloud” brochures skip. No combination of open source projects reproduces 100 % of the public cloud for you, and not for lack of maturity, but by design.

What you will not replicate is, first, instant hardware elasticity: you still buy the peaks up front. Second, the depth and breadth of the managed services catalogue, hundreds of services with a service level agreement and an on-call team behind them, which no internal team matches. Third, the global multi-region network the cloud gives you almost for free. And fourth, that model of transferred operational responsibility: with open source, operations come back home.

The correct reading is not that repatriating is a bad idea, but that you swap a predictable operating expense and a delegated responsibility for a cheaper capital investment and a responsibility of your own. For sustained base load it works out. For everything else, think about it.

When NOT to repatriate

This section has to be here, because repatriation badly applied is expensive. There are three clear signs that the cloud is still your right answer.

The first is a workload with extreme peaks or very variable traffic. Most inference deployments operate between 40 and 65 % GPU utilisation, and below roughly 70 % sustained usage the cloud beats you on total cost. If your workload has long troughs, you are buying iron so it can sleep.

The second is uncertainty or the start of a project. The break-even point in favour of buying only applies to you when utilisation is consistently high and, above all, measured and not projected. Buying GPUs against an optimistic forecast is the fastest way to end up with a warehouse of idle silicon. Sizing it properly is worth a whole article: sizing and justifying the GPU investment.

The third is the lack of an operations team. The total cost of ownership on-premise includes between half and one full-time engineer per GPU server. Without that team, your saving is illusory, because the on-call the cloud included is now covered by nobody. a16z itself frames its thesis as a hybrid approach, not as “leave the cloud” at all.

The numbers, to ground it

So as not to leave you with the discussion in the abstract, here are the orders of magnitude, triangulating two sources with opposite biases, a cloud provider and a neutral FinOps tool. An H100 in the cloud runs you between 2.90 and almost 7 dollars per GPU-hour depending on provider; bought, an H100 card is around 31,000 dollars and an eight-way system between 250,000 and 320,000. At full utilisation, a rented H100 passes its purchase price in less than a year, and an eight-way system with second-hand hardware can pay for itself in about seven months. The three-year total cost of an eight-H100 system, counting staff, colocation, energy and cooling, is estimated at between 700,000 and 950,000 dollars, and the threshold where on-premise beats the hyperscalers is around 80 % sustained utilisation.

The two sources disagree on the exact number, as you would expect, but they agree on the deciding factor: sustained utilisation, measured and not projected. The full working of this calculation is in the TCO of an on-premise GPU cluster. Here the conclusion is enough for you: for continuous base load the numbers work out, and that is exactly the profile of an inference factory.

For your sovereign inference factory

Everything above arranges itself into a practical decision. If you operate an LLM inference platform on your own iron, you are not deciding whether to repatriate (you already took that decision when you built the cluster), but how to rebuild the cloud experience your teams miss.

The sensible sequence starts with the substrate, which you probably already have in the form of RKE2 and storage. It goes on to the provisioning control plane, where Crossplane turns “open a ticket and wait” into a declarative API. It continues with the developer experience, where you choose between the Backstage portal, the KubeVela application model or the portable Score specification, depending on whether your people prefer a web or a CLI. And do not forget the on-premise managed services or the cost allocation, because without a bill your platform loses the discipline the cloud imposed by default.

The next two articles go down into the two layers least covered in this blog: Crossplane for the infrastructure control plane, and KubeVela with Score for CLI self-service centred on the application. With them, the map of this layer is covered, and your own house starts to look like the hotel you left, with the difference that the data stays inside.

See also

Sources