Sovereignty on lo0 — Tech Blog

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

Mon, 31 Aug 2026 09:45:00 +0200

Third and last article in the batch about rebuilding the cloud on-premise. The general map placed the gaps for you and Crossplane covered the infrastructure control plane. Here comes the layer above, the one your developer touches every day: application-centric self-service by CLI.

TL;DR

Crossplane answers “give me a database”, from the infrastructure side. KubeVela and Score answer “deploy my application”, from the workload side, and they do it by CLI, which is precisely the friendly experience Crossplane deliberately does not give you. The two share an article because they cover the same layer from different angles. KubeVela is a complete delivery platform: it uses the OAM model, where your platform team writes some definitions in the CUE language once and the developer deploys their whole application with a fifteen-line file and a CLI (vela up, vela status, vela logs) that feels like Heroku on the outside and is GitOps on the inside. Score is more modest and more portable: a workload specification in a single file that gets translated by CLI into Docker Compose locally and into Kubernetes in production, without running anything in the cluster.

The honest part up front is the limits. Neither of the two appears in the “Adopt” section of the CNCF technology radar from early 2026, where Helm, Backstage and Argo CD do: they are niche against the mainstream. KubeVela version 1.11 is still in alpha, so the stable one is 1.10; its community is concentrated, with around 33 people contributing in 2025. Score is young, its Kubernetes implementation has few stars and its sponsor, Humanitec, was acquired by Akamai. And KubeVela’s power lives in CUE, an uncommon language that is hard to debug. The abstraction is real, but when it breaks, your developer ends up looking at Kubernetes YAML all the same.

The analogy: the order form against the shopping list

Deploying on raw Kubernetes is going to the wholesaler with a technical shopping list: you have to know that you want a Deployment, a Service, a HorizontalPodAutoscaler, an Ingress, a ConfigMap and a ResourceQuota, and how each one gets filled in. It works if you know the warehouse. But your data scientist, who just wants to serve their model, should not have to learn the warehouse.

KubeVela is an order form. Instead of the technical list, you tick “I want to serve this application, with these replicas, with this gateway”, and behind it somebody who does know the warehouse (your platform team) has defined exactly what gets ordered for that to work. The developer fills in the form; the translation into the technical list was done by somebody else, once, for everyone.

Score is the same form but designed so that it works for you in two different shops. You describe your order once and the same sheet serves the market next door (Docker Compose, your laptop) and the wholesaler (Kubernetes, production), without rewriting it. Whatever changes between one shop and the other is supplied by whoever translates, not by you. Both analogies point at the same thing: separating what the developer wants from how it materialises, so that they do not have to learn the warehouse.

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

KubeVela is a CNCF project at incubation level since February 2023. It was born as an evolution of the OAM runtime on Kubernetes, with initial contributions from Alibaba Cloud, Microsoft and Upbound, and it rests on the Open Application Model, the specification Alibaba and Microsoft co-created to describe applications independently of the infrastructure.

Here is a realism check on its community size. The official 2025 retrospective declares seven releases in the year, 124 commits and around 33 people contributing, on a little over 7,600 GitHub stars. It is not a project in decline, but it is a concentrated community, with a growth curve that has flattened compared to its early stage under Alibaba. And an important version warning: the stable branch is 1.10, with regular patches through 2026; 1.11 is still in alpha, so do not rely on its features as if they were available.

The OAM model in five pieces

Your whole application is described in a single YAML file of type Application, with four sections you need to know by name. The components are the artefact you deploy, be it an image, a Helm chart or a service; the traits are operational requirements you hang off each component, such as scaling, the gateway or storage; the policies are global application strategy, such as multi-cluster topology or SLOs; and the workflow is the step-by-step delivery process, which can include a manual approval pause or a notification.

The key to the model is that component and trait types are not fixed: they are programmable modules maintained by your platform team, expressed as ComponentDefinition and TraitDefinition. The developer only consumes a type and its properties; they never see the Kubernetes YAML underneath. That is exactly the “do not expose raw Kubernetes” this is all about: your platform operator writes the definition once, your data scientist writes fifteen lines of Application.

CUE, the power and the toll

The definitions are written in CUE, and here you have KubeVela’s strength and weakness at the same time. CUE lets you parameterise seriously: the definition declares which inputs it exposes to the developer and which Kubernetes resources it renders from them, with access to context variables such as the application name. The real flow starts from an existing YAML, which vela def init turns into a CUE skeleton; you edit it, validate it with vela def vet and publish it with vela def apply, and it becomes available to all your developers.

The toll is that CUE is an uncommon language, with few experts and hard to debug. The cost falls on your platform team, not on the developer, which is the correct distribution, but it is a real cost you have to sign up for before adopting. Nobody arrives already knowing CUE.

The vela CLI: Heroku on the outside, GitOps on the inside

Here is the answer to the question this blog has been dragging along about CLI provisioning. Day to day with KubeVela looks like Heroku: vela up deploys the application from local files, vela status shows you its state, vela ls lists the applications, vela logs follows the logs, vela exec runs a command inside the container without you knowing the pod name, and vela port-forward opens a tunnel for you. You even have vela dry-run, which renders the Kubernetes resources without applying them, so you can see what is going to happen.

The honest qualification is that it is “imperative on the surface, declarative underneath”. When you run vela up you are not executing an imperative command like heroku scale, you are applying a declarative Application object that a controller reconciles. The CLI imitates Heroku’s comfortable flow, but the underlying model is GitOps, which means it fits with your Flux stack through the corresponding addon, instead of replacing it.

Multi-cluster, which is what matters to you

If you have a platform with several sites (site01 and site02, in my case) the multi-cluster part is the deciding one. KubeVela orchestrates from a central cluster and only the already rendered resources reach the managed clusters, with centralised governance. A topology policy decides the destination by cluster list or by label selector (by region, for example); an override policy adjusts images, replicas or traits per cluster without touching the component; and the deployment step in the workflow distributes, with the option of a phased rollout: first to the local cluster, then a manual approval gate, and only then to production. It is the kind of governance you have to build on-premise and that the cloud served you ready-made.

VelaUX, the lightweight portal thrown in

KubeVela comes with an optional web console, VelaUX, which you switch on as an addon and which gives you a dashboard, project management, environments and application lifecycle. If you want CLI first it is optional, more useful for giving operators visibility and access control than as the main route. It is a lightweight portal “thrown in” that does not aim to compete with Backstage, but it can save you from setting one up if your needs are modest.

Score: describe the workload once

What it is and what it solves

Score is an open workload specification, agnostic of platform and environment. The idea, in its own formulation, is that you define the workload once and use a Score implementation to translate it to several platforms: Docker Compose, Kubernetes, Fly.io or Cloud Run. The problem it attacks is concrete and familiar: the divergence between what you run on your laptop with Docker Compose and what gets deployed in production with Kubernetes, two descriptions that drift apart and cause you the classic “it worked on my machine”.

In a score.yaml file the developer declares their containers, the service ports and the resources they need as dependencies (a database, a DNS record, a route), and the environment-specific parameters are injected at deploy time. Score entered the CNCF sandbox in July 2024. It was created and donated by Humanitec, a developer platform company, some eighteen months before it joined the foundation.

The two implementations

Score does nothing on its own: it is a specification that needs an implementation to translate. You have two official ones. score-compose generates Docker Compose for local development, with init and generate commands, and provisions on the order of fourteen resource types, from Postgres or Redis to S3 storage or queues. score-k8s generates Kubernetes manifests ready for kubectl apply, with flags for the image, the namespace and the provisioners. Humanitec’s commercial CLI, humctl, also supports Score natively.

The most important architectural fact is that Score runs nothing in the target cluster. It is purely CLI translation: it generates manifests for you and steps aside. There is no controller, no CRD, no reconciliation. That is at once its virtue (it is lightweight, portable and without runtime lock-in) and its limit (it neither orchestrates nor deploys, it only translates).

Score against KubeVela and Helm

Score’s own official positioning clarifies the layers, and I will pin it down for you because they are easy to confuse. Score is the workload specification layer, developer-centric: it describes a workload and its dependencies and gets translated to N platforms by CLI, but it does not orchestrate or deploy. KubeVela is a complete continuous delivery platform, application-centric: it orchestrates multiple components, applies policies and workflows, does multi-cluster, and for that it needs its controller in the cluster; it is, in its own words, far richer in features than Score. Helm is the packaging and rendering of manifests, without the developer-centric abstraction or the portability to Compose.

The clean synthesis is that they are complementary layers, not substitutes: Score generates, Helm or the manifests package, KubeVela orchestrates and deploys. If they are presented to you as rivals, somebody has confused the layers.

Where each one fits: the cross-cutting comparison

With Crossplane already in hand from the previous article, the comparison of the three self-service pieces sorts itself by the question each one answers.

Score and KubeVela answer at the application level: the workload a developer deploys, “deploy this app”. Crossplane answers at the infrastructure level: the resource a developer asks for, “give me a managed Postgres”. And Backstage answers at the portal level: the web catalogue and the golden paths, the discovery layer that neither deploys nor provisions by itself, but orchestrates the ones below.

The interesting part is that they combine. A score.yaml can declare its resources and delegate the provisioning of those resources to Crossplane through a provisioner, while KubeVela orchestrates the application deployment and Backstage acts as the façade. It is not an either-or; it is a stack where each layer does its job. If your team is small, you pick a single piece and that is that; the complete stack is for mature platforms.

The fit with inference: a pattern, not an official recipe

Just as with Crossplane, honesty is called for here: there is no official, maintained KServe definition in KubeVela’s default catalogue, nor an inference resource type in Score. What follows is a design pattern with pieces confirmed separately, not a documented recipe.

The KubeVela pattern would be that your platform operator defines once, in CUE, a ComponentDefinition called for example inference-service, which underneath renders a KServe inference service with vLLM runtime, with the GPU limits, the namespace quotas and the autoscaling already wired in. Your data scientist then writes a fifteen-line Application with the inference-service type and its properties (the model, the replicas), without touching KServe, without knowing the GPU quotas or the runtime class, and operates with vela up, vela status and vela logs. Multi-site is solved for you by the topology policy, which decides which GPU cluster it lands on, and the override policy, which adjusts replicas or model per site, with a manual approval in the workflow before production.

The Score alternative would be a score.yaml with a resource of type inference-endpoint that score-k8s, with a custom provisioner, emits as a KServe inference service. More portable, but today it requires that provisioner of your own, because KServe is not a default type. In both cases, the abstraction layer rests on pieces that are already standard (KServe moved to CNCF incubation at the end of 2025 and vLLM is its reference generative runtime), but the inference abstraction is built by you. Take it as a proposal, not as a proven case.

A note for you: in 2026 KServe introduced a type dedicated to large models, LLMInferenceService, still in alpha and with vLLM as its runtime, which is the brick a KubeVela definition or a Score provisioner would render underneath. And at the infrastructure layer, not this one, the Crossplane team published Modelplane, an inference control plane made of compositions that serves vLLM directly. Neither of the two is a KubeVela or a Score type, but both confirm for you that the brick underneath is standardising, which is what makes putting a self-service façade on top viable.

Caveats, no frills

This is the section that separates the article from the brochure, and with these two projects you need it especially.

Adoption is niche, and there is a hard figure. The CNCF technology radar from early 2026, covering more than 400 developers, places Helm, Backstage and Argo CD in “Adopt”. Neither KubeVela, nor Score, nor Crossplane get a mention. They are niche tools against the mainstream, and choosing them means betting on an adoption curve that may not take off.

Score is young and its sponsor changed hands. Its Kubernetes implementation has few stars, the specification is at 0.x versions, and Humanitec, which sponsors it, was acquired by Akamai in 2025, so you have to watch the continuity of the investment in the open source part.

KubeVela has a concentrated community and v1.11 in alpha. The 33 people contributing in 2025 and the flat star curve are a sign of a plateau. Do not build on 1.11 features until it is stable.

The CUE curve is real. KubeVela’s power lives in CUE definitions, and CUE is hard. The cost falls on the platform, but it exists.

And the usual risk with abstractions: the leak. Both KubeVela and Score add an indirection over Kubernetes and KServe. When something fails underneath, the abstraction cracks and your developer ends up looking at the KServe YAML anyway, plus lock-in on your own definitions. An abstraction that does not hold up under failure hands you back exactly the complexity it promised to hide.

For your sovereign inference factory

The practical decision sorts itself by the profile of your users. If your people live in the CLI and think in applications, KubeVela gives you the Heroku-style experience on your own iron, with real multi-cluster, in exchange for your platform team learning CUE. If what hurts most is the divergence between local and production, Score solves that with the bare minimum, without putting one more runtime in the cluster. And if you want both plus a portal, the Score, Crossplane, KubeVela and Backstage stack exists, but it is for a platform that already serves many teams, not for the start.

The sobriety criterion is the same as throughout this batch: these tools plug a real gap left by the cloud you walked away from, but each one is one more system to operate. For two teams who know each other, good Helm and a tidy GitOps repository do the job without CUE or definitions of your own. Application-centric self-service starts paying you back when the number of people deploying exceeds the number of people who know Kubernetes, which is precisely the moment an inference factory stops being a project and starts being a platform.

That closes the batch. The map drew the gaps of leaving the cloud, Crossplane covered the infrastructure control plane, and KubeVela with Score cover the face the developer touches. Your own house now looks quite a lot like the hotel, and the data stayed inside.

Sources

KubeVela, State of KubeVela 2025 — https://kubevela.io/blog/2025/12/20/state-of-kubevela-2025/
CNCF, KubeVela brings software delivery control plane capabilities to the CNCF incubator — https://www.cncf.io/blog/2023/02/27/kubevela-brings-software-delivery-control-plane-capabilities-to-cncf-incubator/
KubeVela Docs, Core Concept — https://kubevela.io/docs/getting-started/core-concept/
KubeVela Docs, vela CLI reference — https://kubevela.io/docs/cli/vela/
KubeVela Docs, Custom Component with CUE — https://kubevela.io/docs/platform-engineers/components/custom-component/
KubeVela Docs, Multi-cluster delivery — https://kubevela.io/docs/case-studies/multi-cluster/
KubeVela Docs, VelaUX addon — https://kubevela.io/docs/reference/addons/velaux/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Score Docs, Overview — https://docs.score.dev/docs/
Score, Score vs the Open Application Model and KubeVela — https://score.dev/blog/score-vs-open-application-model-kubevela/
GitHub, score-compose — https://github.com/score-spec/score-compose
GitHub, score-k8s — https://github.com/score-spec/score-k8s
CNCF, KServe becomes a CNCF Incubating Project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
CNCF y SlashData, Technology Radar Q1 2026: Platform Engineering — https://www.cncf.io/announcements/2026/03/24/cncf-and-slashdata-report-finds-platform-engineering-tools-maturing-as-organizations-prepare-for-ai-driven-infrastructure/

Crossplane: the control plane that turns your cluster into a private cloud API

Mon, 31 Aug 2026 09:30:00 +0200

Second article in the batch about rebuilding the cloud experience on-premise. The general map placed the gaps for you; this one goes into the provisioning control plane, that API which on-premise does not come for free. The third moves up a layer, to application-centric self-service with KubeVela and Score.

TL;DR

Crossplane solves the “open a ticket and wait” gap for you. In the cloud you ask for a database through an API and it appears with permissions, backups and monitoring; on-premise you do that provisioning by hand or with scripts. Crossplane brings the Kubernetes model of declarative APIs to infrastructure: your platform team defines an API of its own (a database, an inference endpoint) and the developer consumes it with a kubectl apply of a few fields, while controllers inside the cluster reconcile the desired state against reality and correct drift continuously. It graduated from the CNCF in November 2025, with more than three thousand contributors and adopters such as Nike, SAP, IBM or NASA’s science cloud.

The difference from Terraform is one of nature, not of features: Terraform applies a plan on demand and leaves; Crossplane is a control plane that lives inside the cluster and never stops reconciling. That gives you drift self-correction and native GitOps, and in exchange it takes away the safety net of plan and forces you to operate the control plane itself, with providers that create hundreds of resource definitions and state that lives in etcd. And a warning about something awkward up front: Crossplane is not an imperative aws ... style CLI, it is declarative first. On the fit with GPU inference, which until recently was only a pattern with no public case, you now have a real reference, Modelplane, from the Crossplane team itself, although at version 0.1 and composing vLLM directly without going through KServe. I go into the detail at the end.

The analogy: the thermostat against the light switch

Think of it this way: Terraform is a light switch. You flip it, the light comes on, and it stays there until you touch it again. If somebody else changes it, the light obeys whoever touched it last, and you do not find out until you go and look. It is provisioning by one-off action: you run it, it applies, it finishes.

Crossplane is a thermostat. You do not tell it “turn the heating on”, you tell it “I want twenty-one degrees”, and from then on a mechanism watches the temperature and acts on its own, continuously, to hold it. If somebody opens a window and the thermometer drops, the thermostat reacts without you intervening. That is continuous reconciliation: you declare the desired state once and a controller takes care of reality not straying from it.

The analogy also warns you about the cost. A thermostat is more convenient than a switch, but it is one more device you have to install, power and fix when it fails, and if you program it badly it adjusts the temperature to something you did not want, without asking. A switch almost never breaks and you always know what state it is in. Crossplane gives you the convenience of the thermostat in exchange for you operating the thermostat.

What Crossplane is and what state it reaches 2026 in

Crossplane was born at Upbound in 2018 and followed the CNCF maturity path without shortcuts: sandbox in 2020, incubation in 2021 and graduation on 6 November 2025. Graduation matters because it is the seal the CNCF reserves for projects it considers production-ready, the same level as Kubernetes or Prometheus. The figures that came with the announcement are solid: more than three thousand contributors from more than 450 organisations, more than a thousand pull request authors (which puts it in the top 10 % of the foundation’s projects by that metric), more than a hundred releases and two completed security audits. Among the more than seventy public adopters you have Nike, Nokia, Grafana, NASA’s science cloud, SAP and IBM.

The project’s official description is “the cloud native control plane”, and that phrase sums up exactly what it does. The main vendor behind it is still Upbound, which at graduation framed the project’s future towards “AI-native” infrastructure, a point I come back to when talking about inference, with the caveat that it is a vendor’s positioning, not a proven case.

The release cadence is quarterly, with a nine-month support window and three actively maintained versions at any time. As of writing, the v2 branch is the current one, with v1.20 as the last close of the v1 branch.

The v2 turn (which changes how you think about it)

Crossplane 2.0 reached general availability in August 2025 and brings four structural changes you need to know about, because a good deal of the old material describes a model that is no longer the recommended one.

The first is that resources are now namespaced by default. Both composite and managed resources, which used to be cluster-scoped, now live in a namespace, which enables fine-grained access control and natural multi-tenancy, exactly what you need in a sovereign platform with several teams.

The second is that a composition can include any Kubernetes resource, not only Crossplane resources. Previously a composition was limited to Crossplane’s own managed resources; now it can include Deployments, Services or third-party CRDs, so you can compose abstractions mixing application and infrastructure in a single object. This change is what makes the inference pattern you will see viable.

The third is that Claims disappear. In the v1 model there was a duality between the composite resource, cluster-scoped, and the Claim, its namespaced face which the developer consumed. Now that the composite resource is itself namespaced, that duality is redundant: the developer consumes the composite resource with no intermediary. If you come across documentation talking about Claims, it belongs to the earlier model.

The fourth is a new type, Operations, for one-off or scheduled operational tasks that run a pipeline of functions to completion, like a Kubernetes Job but inside the Crossplane model.

There is one removal that deserves its own paragraph, because it is the most surprising: in v2 native patch-and-transform was withdrawn, the classic patching mechanism of compositions. It was not relegated, it was removed. Everything now goes through composition functions, and I get to that in a moment. Migration from v1 to v2 is mostly non-breaking and the v1 APIs still work, so adopting v2 is optional, but the new model is the one you have to learn.

The model, piece by piece

Crossplane has dense jargon and I will take it apart calmly, because once the pieces are placed the whole thing is coherent.

A provider is a package that installs controllers and resource definitions to talk to an external API, be it AWS, GCP, a remote Kubernetes cluster, Helm or Terraform. It is installed declaratively, by applying a Provider object. The providers for the big cloud providers are generated with Upjet, a framework that produces the code from the Terraform providers, which explains why there are so many of them and also why some are enormous.

A managed resource is the one-to-one representation, inside Kubernetes, of a resource from the external provider: a bucket, an instance, a database. It is the unit the controller reconciles. When you create a managed resource of type bucket, the provider’s controller creates the real bucket and then watches that it keeps existing with the configuration you declared.

On those bricks you build the abstraction, which is where the real value is, with three concepts:

The CompositeResourceDefinition, or XRD, defines the schema of your own API: which fields the user can ask for. It is what turns “a database” into a resource type with its parameters.

The Composition is the template and the logic that, given one of those resources, materialises the N managed resources needed underneath. The documentation says it bluntly: a composition is a pipeline of composition functions.

The Composite Resource, or XR, is the concrete instance: the object the developer creates with kubectl apply, with the few fields the XRD exposes.

The complete pattern is this: your platform operator publishes the abstraction (the XRD plus the Composition, which say “this is what a database means here”), and the developer only applies a composite resource with four fields. How it is built underneath stops mattering to them, just as in the cloud you do not care how AWS assembles your managed database.

Composition functions: the real logic

The composition mechanism in v2, and the only one, is composition functions. A composition is a pipeline of functions that Crossplane invokes over gRPC; each function receives the observed state, the desired state, its input and a shared context, and chains its output to the next one. The YAML in the examples is illustrative, but underneath it is not processed as YAML: it is executed as functions.

The available functions cover several styles. You have function-patch-and-transform, which reimplements the old model now as just another function. There is function-go-templating for Helm-style templates, function-kcl for the KCL language, function-python for logic in Python and function-cue for CUE.

The reason for the change is what you want to understand. Classic patch-and-transform had no loops or conditionals, so an operation as simple as transforming an array of strings turned into dozens of lines of patching, and the community went as far as calling it outright bad. Functions give you full programming logic (loops, conditionals, external calls) in a real language. The honest flip side is that the complexity does not disappear: it shifts from “writing verbose patches” to “writing and operating functions”, which is a different kind of work, not less work.

Crossplane against Terraform, OpenTofu and Pulumi

This is the comparison everybody makes, and I will make it properly because the difference is one of category. According to Pulumi’s own documentation, which compares honestly, Crossplane is a declarative control plane with continuous reconciliation inside Kubernetes, while Terraform and Pulumi are infrastructure as code executed one-off, client-side and on demand.

The practical consequences line up along a few axes. Faced with configuration drift, Crossplane corrects it continuously and automatically, while Terraform and Pulumi only detect it when you run a refresh. Crossplane needs a cluster to run on, because it is controllers; Terraform and Pulumi do not. Crossplane is GitOps-native, applied with kubectl and fitting with Flux without an adapter; Pulumi needs its Kubernetes operator for the same thing. And as for language, Crossplane uses YAML and CRDs with logic in the functions, while Pulumi uses general-purpose programming languages.

The point most often forgotten is that you do not have to choose one or the other. There is a Terraform provider (and an OpenTofu one, recommended for new versions because the Terraform one was frozen at the last freely licensed version) that runs HCL inside Crossplane, wrapping your existing modules as a managed resource. You can reuse your whole investment in Terraform and still gain continuous reconciliation for whatever needs it. If you already have Terraform in production, this hybrid pattern is usually your sensible route, not the rewrite.

On comparative adoption you have to be sober: there is no official, reliable “Crossplane against Terraform” figure, and the most repeated documented criticism is that Crossplane lacks the mature module ecosystem Terraform has been accumulating for years. That is a real qualitative difference, not a detail.

The awkward question: is it an AWS-style CLI?

Since on this blog I have been pulling on the thread of CLI provisioning, I will answer directly: no. Crossplane does not offer you an imperative aws s3 create-bucket style experience. It is declarative first. You provision with a kubectl apply of a resource, and the crossplane binary is there to build, package, test and debug, not to “create a bucket” in one go.

The subcommands make that clear. crossplane render renders a composition and its functions locally, so you can test without a cluster. The crossplane xpkg build, push, install family manages the OCI packages of providers, configurations and functions. And crossplane beta validate, trace and top validate schemas, draw the resource tree and show consumption inside the cluster. It is the toolbox of whoever builds the platform, not of whoever consumes it day to day. Upbound’s up CLI adds more, but oriented towards its commercial product.

The conclusion matters for setting expectations: the imperative AWS-console-style experience, if you want it, gets built on top of Crossplane, with Backstage, a portal or your own wrapper. Crossplane gives you the declarative API; the friendly face you put on separately. If what you are after is an imperative provisioning CLI, you fit better with the model of KubeVela and Score, which is precisely the subject of the next article.

On-premise providers: provisioning without touching AWS

For a sovereign platform your question is what you can provision without setting foot in a public cloud, and the answer is more than it seems, with uneven maturity.

The most versatile is provider-kubernetes, which manages any Kubernetes object in remote clusters through an Object resource. It is the basis of the multi-cluster on-premise pattern: from a central cluster you materialise resources in your secondary RKE2s. Alongside it, provider-helm installs and manages Helm charts declaratively through a Release resource, letting you deploy things like vLLM or KServe as part of a composition.

For the identity and storage part you have concrete pieces: provider-keycloak configures Keycloak realms, clients and roles declaratively, useful for the sovereign IAM we already set up in the article on authentication with Keycloak; and provider-minio manages buckets and policies of on-premise S3-compatible storage. For vSphere you do not have a first-class native provider, but you cover it by generating one from the Terraform provider, the usual pattern for VMware infrastructure.

An honest warning: for KubeVirt, Harvester or CloudNativePG there are no mature, first-class native providers; in practice you manage them by applying their CRDs through provider-kubernetes or by wrapping their Terraform. It works, but it is not the polished experience of a dedicated provider. With the combination of provider-kubernetes, provider-helm, provider-minio and provider-keycloak, plus provider-terraform for whatever is missing, you can assemble a one hundred per cent on-premise self-service catalogue, with the caveat that maturity varies piece by piece.

The fit with GPU inference: from pattern to Modelplane

This is the section that in the first version of this text carried the most caution, because there was no public case of Crossplane orchestrating GPU and inference and it all remained a design pattern. Halfway through 2026 that changed, and the case was published by the very team that makes Crossplane.

Modelplane is an open source control plane for AI inference, Apache-licensed and with a first version 0.1 from June 2026, and the interesting thing is how it is built: entirely on Crossplane compositions and composition functions in Python, with no controllers of its own. It composes vLLM directly, with the vllm/vllm-openai image, without going through KServe, and its self-service API for the ML teams is a ModelDeployment resource where you declare the model, the engine, the topology and the hardware. Underneath, a ModelReplica resource represents each replica, and the platform team declares its GPU pools with InferenceCluster and InferenceClass resources, using an attribute model and CEL expressions taken from Kubernetes dynamic resource allocation. Its most advanced piece is a two-level fleet scheduler written as a pure function, which places replicas across the whole infrastructure and delegates to each cluster’s scheduler. It is proof that the pattern works, with the warning that it is a version 0.1 under open development, not something to put into production blindly, and that adopting it ties your platform to its resource types.

The other recent enabler is in KServe, which introduced a type dedicated to large models, LLMInferenceService, still in alpha and built on the llm-d project, with prefill and decode separation and multi-node execution, and which runs vLLM as its runtime. It is different from the classic InferenceService, reserved for predictive ML. That new type is your ideal brick if you prefer composing KServe instead of bare vLLM.

With that, your own golden path stops being a hypothesis. You define an XRD, for example InferenceEndpoint, with fields such as the model, the number and type of GPU and the replicas. Its composition, taking advantage of the fact that in v2 it can include any Kubernetes resource, materialises in a single pipeline the team’s namespace, the quota for nvidia.com/gpu and a KServe LLMInferenceService pointing at the model with vLLM runtime, and the developer only applies the InferenceEndpoint with four fields, knowing nothing about KServe or the quotas underneath. KServe supports vLLM and GPU natively, as we saw in the article on KServe.

The honest reading is that you are at the frontier, not on well-trodden ground. Outside Modelplane, made by Upbound, there is still no body of independent cases assembling Crossplane with vLLM, and the KServe type is in alpha. It is a coherent architecture, with a real reference backing it and plenty of room for you to be among the first to document it on your own iron.

What it costs you to operate your own control plane

No tool that gives you back control is free, and Crossplane has well-documented costs you need to budget for beforehand, not discover afterwards.

The learning curve is steep, acknowledged even in the coverage of its graduation. Adopters report debugging difficulties when a composition or a provider misbehaves, and they miss Terraform’s module ecosystem.

Compositions tend towards complexity. There are practitioners describing definitions spanning thousands of lines with poor navigation, and the absence of an integrated testing framework. Functions improve this, but as I said, they shift the complexity rather than remove it.

There is no plan. Unlike Terraform’s plan and apply cycle, Crossplane applies changes immediately. You lose the safety net of reviewing what is going to happen before it happens, with the risk of modifications you did not want.

Operating the control plane itself costs. Big providers create hundreds of resource definitions (the GCP one reached 347), with a risk of API server unavailability and of leaving your cluster hanging. State lives in etcd, which complicates disaster recovery and migrating control between clusters. And there is one specific danger: accidental deletion of cloud resources when upgrading XRDs, providers or compositions, because of the ownership Crossplane keeps over what it created.

Some have decided not to adopt it. The Masterpoint team ruled it out for the lack of data sources and ignore_changes, for incomplete coverage of AWS services, and for the cost of abandoning their accumulated Terraform experience. It is a legitimate decision, and quoting it to you is part of an honest article.

How it lives alongside what you already have

If you already run a platform, your question is not Crossplane yes or no, but Crossplane with what. Three relationships matter to you.

With Backstage they are complementary, not rivals. Backstage is the portal and the templates that generate manifests or pull requests, day zero; Crossplane provisions and reconciles the real state, day two. The usual pattern, which some call the golden triangle, is Backstage as the interface, Crossplane as the control plane and Argo CD or Flux as GitOps. Each one does a layer.

With Flux, which you probably already use, the relationship is additive. Flux reconciles manifests towards the clusters, but it does not give you a platform abstraction API: it does not know what “a database” is, it only applies the YAML you hand it. Crossplane adds exactly that abstraction layer, and the two coexist, with Flux deploying the composite resources and Crossplane materialising them. If you already have Flux, Crossplane’s value is precisely that self-service API layer, in exchange for the cost of operating the control plane I described above.

With KubeVela and Score, the subject of the next article, the boundary is one of layer: Crossplane answers “give me a Postgres”, from the infrastructure side; KubeVela and Score answer “deploy my application”, from the workload side. They do not compete, they stack.

For your sovereign inference factory

The practical decision boils down to this. Crossplane makes sense when you want to expose a self-service API of your own over your on-premise infrastructure and you are willing to operate a control plane to get it. If your platform is used by two teams who know each other and you provision with a tidy Flux repository, Crossplane is cost without return for now. The trigger is scale: when manual provisioning turns into a ticket bottleneck, the declarative API starts paying back its cost.

Your sensible entry route is not rewriting everything. It is starting with a provider that already serves you (provider-kubernetes and provider-helm cover a lot), wrapping your existing Terraform with the corresponding provider instead of throwing it away, and modelling a single abstraction of clear value, probably the InferenceEndpoint we discussed, as a golden path. From there, you grow on demand. And always with a clear account of having traded the safety net of plan and a handful of scripts for a thermostat you have to keep switched on.

The next article moves up a layer, from the infrastructure resource to the application workload, with KubeVela and Score, which is where the friendly CLI that Crossplane deliberately does not give you finally appears.

Sources

CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
Crossplane Blog, Announcing Crossplane’s CNCF Graduation — https://blog.crossplane.io/crossplane-cncf-graduation/
Crossplane Blog, Announcing Crossplane 2.0 — https://blog.crossplane.io/announcing-crossplane-2-0/
Crossplane Docs, What’s New in v2 — https://docs.crossplane.io/latest/whats-new/
Crossplane Docs, Compositions — https://docs.crossplane.io/latest/composition/compositions/
Crossplane Docs, Release Cycle — https://docs.crossplane.io/latest/learn/release-cycle/
Crossplane Docs, Crossplane CLI — https://docs.crossplane.io/latest/cli/
Upbound, Crossplane Graduates From CNCF, Upbound Redefines AI-Native Infrastructure — https://www.upbound.io/blog/crossplane-graduates-from-cncf-upbound-redefines-ai-native-infrastructure
InfoQ, Crossplane Reaches Production Maturity by Graduating CNCF — https://www.infoq.com/news/2025/11/crossplane-grad/
Pulumi Docs, Pulumi vs. Crossplane — https://www.pulumi.com/docs/iac/comparisons/crossplane/
Upbound Marketplace, provider-terraform — https://marketplace.upbound.io/providers/upbound/provider-terraform/latest
GitHub, provider-kubernetes — https://github.com/crossplane-contrib/provider-kubernetes
GitHub, provider-helm — https://github.com/crossplane-contrib/provider-helm
GitHub, provider-keycloak — https://github.com/crossplane-contrib/provider-keycloak
VSHN, provider-minio — https://github.com/vshn/provider-minio
Masterpoint, Crossplane: Why it Didn’t Work for Us — https://masterpoint.io/blog/passing-on-crossplane/
CECG, Crossplane: the good, the bad and the ugly — https://www.cecg.io/blog/crossplane-the-good-the-bad-the-ugly
Taloflow, Backstage vs Crossplane for Platform Engineering — https://www.taloflow.ai/guides/comparisons/backstage-vs-crossplane-platform-engineering
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/
Modelplane — https://modelplane.ai · https://github.com/modelplaneai/modelplane
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview

From public cloud to private: what you really lose and which projects fill the gap

Mon, 31 Aug 2026 09:15:00 +0200

With this one I open a run about the biggest gap a sovereign platform leaves you: how to give your teams the self-service experience of a public cloud, without the public cloud. Here you have the map of the problem; the next two go down into the detail of the tools, Crossplane as the infrastructure control plane and KubeVela with Score as application-centric self-service.

TL;DR

Leaving the public cloud has a solid business case, and I am not making it up. 37signals declared savings of close to two million dollars in 2024 after repatriating, the classic a16z analysis puts cloud spend at around half of the cost of revenue of a software company, and the Barclays CIO survey of 2024 recorded the highest repatriation intent in its history. If your workload is a predictable, sustained base (which is exactly the continuous LLM inference this blog is about) the numbers work out, and in Europe the sovereignty reasons pile on top: the Data Act, the tension with the US CLOUD Act, and that dependence on three hyperscalers that control around 70 % of the European cloud market.

But the problem is not the compute. Your on-premise GPU rack serves tokens just as well as a rented one. What the cloud was selling you and you now have to rebuild is something else: the elasticity of asking for a hundred machines at ten and handing them back at eleven, the catalogue of hundreds of managed services with an on-call team behind them, the unified provisioning API with fine-grained permissions, the multi-region network, the bill that arrives itemised, and that model where operating all of it was somebody else’s problem. All of that you rebuild piece by piece with open source projects, and the rebuild has a cost that the savings figures do not usually tell you about. What follows is the map of what you lose, which project fills each gap, and where a gap remains that nothing will close completely.

The analogy: leaving the hotel to set up your own house

Think of it like this: living in the public cloud is living in a hotel. You pay a lot per night, but you do not change light bulbs, reception is open at three in the morning, if you need another room you ask for it and it appears, and when you leave you take nothing with you to maintain. Setting up your own house works out much cheaper per square metre if you are going to stay for years, and on top of that you decide who comes in and where your things are kept. The problem arrives the day a pipe bursts: then you do not call reception, because reception is you.

Repatriation is that move. The savings are real and so is data ownership, but what in the hotel came included in the price of the night is now your to-do list: the plumbing is your network, the boiler is your storage, the concierge who got you anything is the catalogue of managed services you no longer have, and the itemised bill they slid under your door is a cost model you have to build, because electricity and depreciation do not arrive broken down per tenant. The house pays off, but only if you count what it costs to furnish it and fix the pipes, not just the price per square metre against the hotel.

The rest of the article walks through that to-do list and tells you, for each item, which open source tool does the job the hotel used to do for you.

Why this conversation belongs to 2026, not to always

Repatriating is nothing new, but in 2026 three things come together that make it urgent if you run a European AI platform.

The first is economic, and Andreessen Horowitz put it on the table in 2021 with The Cost of Cloud, a Trillion Dollar Paradox. Their thesis, which you should read knowing that a16z has interests in the ecosystem it analyses, is that cloud spend averages around 50 % of the cost of revenue in a software company, and that recovering half of that spend would free up margins the market penalises. The cleanest case, and one you can verify, is 37signals: DHH published that their cloud bill dropped from 3.2 to 1.3 million dollars a year after repatriating, with an initial investment of about 700,000 dollars in Dell servers. Now the honest caveat, which the specialist press itself underlines: those savings figures did not include the extra operations staff, nor the energy, nor the cooling. I come back to it in the hidden costs section.

The second is declared intent. The Barclays CIO survey from the first half of 2024 recorded that 83 % planned to repatriate some workload to private cloud or on-premise in the following twelve months, the highest reading in the series. But read that 83 % precisely: it is the proportion of managers who plan to move some workload, not the percentage of workloads or of spend that moves. Flexera, in its 2025 report, gives you the more grounded figure: large companies have repatriated around 21 % of the workloads they had in the public cloud, and cost saving is priority number one for almost 60 %.

The third is regulatory and European, and it is the one that turns repatriation into something more than a FinOps decision. The European Union Data Act came into force in January 2024 and has applied since September 2025; it mandates portability between providers and, from January 2027, bans the exit fees that today chain you to your provider. And underneath it beats a deeper legal tension: the US CLOUD Act allows United States authorities to demand data from an American provider even if it is stored in Frankfurt, which is exactly the argument for why a European region of a hyperscaler does not give you sovereignty. With AWS, Microsoft and Google controlling around 70 % of the European cloud market, the conversation about sovereign cloud stopped being theoretical. I developed this already in sovereign on-premise versus hyperscalers.

What you lose when you leave the cloud, specifically

Let us get to the core, ordered by what is going to hurt you most. And one thing before starting: none of these gaps is about compute. They are all about what the cloud put around the compute.

Hardware elasticity disappears. On-premise, the GPU you did not buy does not exist. The cloud bills you by the hour precisely because it takes on the risk of idle capacity: it lets you ask for a peak and hand it back. With your own iron, you buy the peak up front and pay for it whether it is in use or not, so your profitability depends on high sustained utilisation. Below a certain usage threshold, the cloud beats you again. This is the irreducible gap, the one no tool fills, and if you repatriate a workload with extreme peaks you are making the wrong decision.

Managed services become your operations. Every box you took for granted (the database, the queue, the object storage, the load balancer, DNS, the secrets manager) stops being a checkbox you tick and becomes a system you install, update and watch yourself. The 37signals case teaches it well: getting out of S3 object storage was the last and hardest part, with petabytes of data and a contract they dragged along for years. Your managed Postgres turns into operating CloudNativePG; your S3 turns into operating MinIO; and so on with every service.

You have to build the provisioning control plane yourself. AWS gives you a unified API where you ask for a resource with fine-grained permissions and it appears. On-premise that does not come for free: you have to build the control plane that translates “I want a database” into real resources. It is exactly the gap that Crossplane fills, and that is why it has its own article.

You replace the managed network piece by piece. The VPC, the security groups, NAT, the load balancer: every network abstraction in the cloud has its on-premise equivalent (Cilium for network policies on eBPF, MetalLB for the load balancer without ELB, Gateway API for L7 ingress), but they are pieces you assemble, not a service you switch on. A good part of this we already saw in the networking vertical with Cilium and eBPF.

Billing stops existing as such. In the cloud the cost reaches you broken down by service and by tag; you have a Cost Explorer that answers “how much does this team cost me”. On-premise there is no natural bill: there is hardware depreciation, an electricity bill and some payroll, and splitting that per workload is a model you have to build. OpenCost helps you with the split per workload inside the cluster, but the total physical cost (depreciation plus energy plus staff) you model separately, as we saw in OpenCost and cost allocation.

You design multi-zone availability yourself. The three availability zones the cloud gives you for free turn into several sites, several racks and several utility feeds that you have to plan. 37signals solved it with two datacenters and replication between them. It is neither free nor automatic.

And above all, the on-call is yours. The cloud was not selling you only compute: it was selling you operations included. When something falls over at three in the morning, in the cloud there is an on-call team at the provider; with open source, that team is you. This is the cost that savings spreadsheets almost never include, and it is the one that decides whether your repatriation was a good idea or a hidden bill.

The map of projects that fill each gap

With the gaps ordered, the catalogue of substitutes reads itself. I group it by layer, from the lowest to the closest to the developer.

In the infrastructure substrate, what replicates EC2 virtual machines, virtual networking and disks, you have OpenStack as the classic standard for a complete private cloud, SUSE’s Harvester as bare-metal hyperconvergence on KubeVirt managed from Kubernetes, and emerging projects such as Spinifex that rebuild an AWS-compatible API on your own iron. It is the heaviest layer and the one I touch least in this blog, because I assume your cluster already exists.

In the provisioning control plane, what replicates the AWS IAM and resource API, there is Crossplane, graduated in the CNCF in November 2025, which brings the Kubernetes declarative API model to infrastructure. Alongside it, Cluster API for declarative cluster provisioning. And so you can see how far this reaches in our own field, the Crossplane team itself published Modelplane in mid-2026, an inference control plane built on compositions that serves models with vLLM. This layer is developed by the Crossplane article.

In the developer self-service experience, what replicates the AWS console and CLI, you have three different approaches you should not confuse. Backstage is the web portal, the catalogue and the templates, and it already has its own article. KubeVela brings self-service to the application model with a friendly CLI. Score defines a workload specification that is portable between environments. KubeVela and Score share the third article because they cover the same layer from angles that complement each other.

In on-premise managed services, CloudNativePG operates Postgres for you with high availability and backups to object storage, MinIO gives you S3-compatible storage, and OpenCost covers that cost allocation you lost with Cost Explorer.

The way to read this map is that none of these pieces is surplus if you want the cloud experience: they are the boxes the cloud ticked for you and that you now tick one by one.

The gap you will not close

I will tell you plainly, because it is the part the “leave the cloud” brochures skip. No combination of open source projects reproduces 100 % of the public cloud for you, and not for lack of maturity, but by design.

What you will not replicate is, first, instant hardware elasticity: you still buy the peaks up front. Second, the depth and breadth of the managed services catalogue, hundreds of services with a service level agreement and an on-call team behind them, which no internal team matches. Third, the global multi-region network the cloud gives you almost for free. And fourth, that model of transferred operational responsibility: with open source, operations come back home.

The correct reading is not that repatriating is a bad idea, but that you swap a predictable operating expense and a delegated responsibility for a cheaper capital investment and a responsibility of your own. For sustained base load it works out. For everything else, think about it.

When NOT to repatriate

This section has to be here, because repatriation badly applied is expensive. There are three clear signs that the cloud is still your right answer.

The first is a workload with extreme peaks or very variable traffic. Most inference deployments operate between 40 and 65 % GPU utilisation, and below roughly 70 % sustained usage the cloud beats you on total cost. If your workload has long troughs, you are buying iron so it can sleep.

The second is uncertainty or the start of a project. The break-even point in favour of buying only applies to you when utilisation is consistently high and, above all, measured and not projected. Buying GPUs against an optimistic forecast is the fastest way to end up with a warehouse of idle silicon. Sizing it properly is worth a whole article: sizing and justifying the GPU investment.

The third is the lack of an operations team. The total cost of ownership on-premise includes between half and one full-time engineer per GPU server. Without that team, your saving is illusory, because the on-call the cloud included is now covered by nobody. a16z itself frames its thesis as a hybrid approach, not as “leave the cloud” at all.

The numbers, to ground it

So as not to leave you with the discussion in the abstract, here are the orders of magnitude, triangulating two sources with opposite biases, a cloud provider and a neutral FinOps tool. An H100 in the cloud runs you between 2.90 and almost 7 dollars per GPU-hour depending on provider; bought, an H100 card is around 31,000 dollars and an eight-way system between 250,000 and 320,000. At full utilisation, a rented H100 passes its purchase price in less than a year, and an eight-way system with second-hand hardware can pay for itself in about seven months. The three-year total cost of an eight-H100 system, counting staff, colocation, energy and cooling, is estimated at between 700,000 and 950,000 dollars, and the threshold where on-premise beats the hyperscalers is around 80 % sustained utilisation.

The two sources disagree on the exact number, as you would expect, but they agree on the deciding factor: sustained utilisation, measured and not projected. The full working of this calculation is in the TCO of an on-premise GPU cluster. Here the conclusion is enough for you: for continuous base load the numbers work out, and that is exactly the profile of an inference factory.

For your sovereign inference factory

Everything above arranges itself into a practical decision. If you operate an LLM inference platform on your own iron, you are not deciding whether to repatriate (you already took that decision when you built the cluster), but how to rebuild the cloud experience your teams miss.

The sensible sequence starts with the substrate, which you probably already have in the form of RKE2 and storage. It goes on to the provisioning control plane, where Crossplane turns “open a ticket and wait” into a declarative API. It continues with the developer experience, where you choose between the Backstage portal, the KubeVela application model or the portable Score specification, depending on whether your people prefer a web or a CLI. And do not forget the on-premise managed services or the cost allocation, because without a bill your platform loses the discipline the cloud imposed by default.

The next two articles go down into the two layers least covered in this blog: Crossplane for the infrastructure control plane, and KubeVela with Score for CLI self-service centred on the application. With them, the map of this layer is covered, and your own house starts to look like the hotel you left, with the difference that the data stays inside.

Sources

Andreessen Horowitz (S. Wang, M. Casado), The Cost of Cloud, a Trillion Dollar Paradox — https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/
Data Center Dynamics, 37signals claims it saved almost 2M USD last year from cloud repatriation — https://www.datacenterdynamics.com/en/news/37signals-claims-it-saved-almost-2m-last-year-from-cloud-repatriation/
The Register, Developer pockets 2M USD in savings from going cloud-free — https://www.theregister.com/2024/10/21/37signals_aws_savings/
Barclays, Technology: 1H24 CIO Survey (PDF) — https://8198920.fs1.hubspotusercontent-na1.net/hubfs/8198920/Barclays_Cio_Survey_2024-1.pdf
Channelnomics, Breaking Down the 83% Public Cloud Repatriation Number — https://channelnomics.com/breaking-down-the-83-public-cloud-repatriation-number/
The New Stack, Updated Stats on Cloud Sustainability, Repatriation and Cost Optimization (Flexera 2025) — https://thenewstack.io/updated-stats-on-cloud-sustainability-repatriation-and-cost-optimization/
Comisión Europea, Data Act | Shaping Europe’s digital future — https://digital-strategy.ec.europa.eu/en/policies/data-act
EU Data Act, Article 29 — Gradual withdrawal of switching charges — https://www.eu-data-act.com/Data_Act_Article_29.html
Kiteworks, How the EU Data Act and GDPR Conflict with U.S. CLOUD Act — https://www.kiteworks.com/gdpr-compliance/eu-data-act-gdpr-cloud-conflict/
Computerworld, EU takes first steps to reduce reliance on US hyperscalers — https://www.computerworld.com/article/4181816/eu-takes-first-steps-to-reduce-reliance-on-us-hyperscalers.html
CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Harvester HCI (SUSE) — https://harvesterhci.io/
CloudNativePG — https://cloudnative-pg.io/
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
Spheron, LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
CloudZero, H100 GPU Cost in 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/

Sovereign on-premise vs hyperscalers: the case with data (cost, energy, performance and sovereignty)

Sun, 14 Jun 2026 05:30:00 +0200

Notation: amounts in euros (N €), decimals with a point; when a source quotes dollars it is marked “USD”. Data centred on Europe and Spain. The dollar symbol is not used (formula delimiter).

What this article covers

Synthesis article of the series (S2), and the heart of the proposal: the case with data for sovereign on-premise against the hyperscalers and the European cloud. Up to here, each track measured its own axis, cost per token (FinOps), goodput (benchmarking), energy and carbon; here the four are crossed, with a fourth dimension no US technical comparison puts up front: data sovereignty. The aim is to answer, with numbers and not with ideology, the question that underpins any investment in an AI platform: serve on your own iron, on a European cloud or on a hyperscaler? And to do it honestly: on-prem does not always win, and saying when it wins and when it does not is what makes the recommendation credible.

The framework: four axes, not one number

The mistake in almost every comparison is reducing the decision to the hourly cost of a GPU. The real decision crosses four axes, and only by seeing them together do you decide well:

Axis	Question	Who measures it in the series
Cost (TCO)	how much does it cost to serve, all in?	FinOps (A2–A8)
Performance	does it meet the SLO, at what goodput?	Benchmarking (B2–B8)
Energy and carbon	how many watts and grams per token?	Energy (C2–C8)
Sovereignty	under which jurisdiction does the data live?	GDPR / EU AI Act / CSRD

The first three are quantifiable and meet in the cost per token; the fourth is a constraint that can rule out an option however cheap it is. The synthesis consists of scoring each option on all four and deciding on the Pareto frontier, not on whichever axis happens to suit.

The cost axis: TCO and break-even

The real on-premise cost is amortised capex plus opex, and its cost per effective hour depends on utilisation:

$$\text{effective cost/GPU-hour (on-prem)} = \frac{\text{annual amortised capex} + \text{annual opex}}{8760 \times \text{utilisation}}$$

This formula is the key to the whole debate: the cost per useful hour of on-prem rises as utilisation falls, because capex is paid whether the GPU is working or idle. The 2026 data:

Data point	Value	Source
On-prem cost 8×H100 (floor, high util.)	~2.83 USD/GPU-hour all-in	Spheron
Annual on-prem cost (floor)	~237,000 USD/year	Spheron
AWS H100 (p5.48xlarge)	4.10–6.88 USD/GPU-hour	Spheron
AWS 8-GPU on-demand annual (100 % util)	287,000–482,000 USD/year	Spheron
Sovereign European cloud (Lyceum/Scaleway)	from 2–2.73 €/GPU-hour, zero-egress	Lyceum, Scaleway

The break-even against AWS on-demand falls around 50–83 % utilisation depending on region and tariff; below ~70 % utilisation, the cloud wins on TCO, and above it, on-prem (Spheron). For very high-utilisation workloads, on-prem pays for itself in less than 4 months (Lenovo).

Three-year TCO: the full calculation for an 8×H100 node

Abstract numbers do not convince a committee; a three-year model with declared line items does. Take a sovereign 8×H100 node in Spain and compare it, for the same work, with AWS and with a European cloud. Declared assumptions: amortisation over 3 years, energy at a solar PPA of 32.5 €/MWh (with grid backup), PUE 1.3, and two utilisation scenarios (50 % and 80 %).

On-premise (own node), annual line items:

Line item	Annual value	Note
Amortised capex (node ~270,000 € / 3 years)	~90,000 €	8×H100 server + network + storage
Energy (≈10.4 kW × PUE 1.3 × 8760 h)	~3,850 € (at 32.5 €/MWh)	with solar PPA; at grid tariff, ~12–18 k €
Operation, cooling, maintenance	~25,000 €	prorated staff, support, spares
Datacenter space (rack, connectivity)	~12,000 €	colocation or own datacenter
Annual total	~131,000 €	independent of utilisation

At a fixed 131,000 €/year, the cost per token depends only on how many tokens you generate, that is, on utilisation:

Utilisation	Useful GPU-hours/year	Cost/1M tokens
50 %	~35,000	~2.9 €
65 %	~45,500	~2.2 €
80 %	~56,000	~1.8 € (with cheap grid, ~1.1 €)

Sovereign European cloud (Scaleway/Lyceum), pay per use: at ~2.2 €/GPU-hour with zero-egress, the cost per token is constant with utilisation (you only pay for what you use): ~1.5–2.2 €/1M tokens depending on model and batching, with no capex and no idle risk.

Hyperscaler (AWS p5), on-demand: at 4.10–6.88 USD/GPU-hour (≈3.8–6.4 €), the cost per token works out at ~2–3.5 €/1M tokens, and on top of that you have to add the egress, quite apart from the fact that for GDPR data the sovereignty axis rules it out already.

The reading of the model is the thesis of the whole of S2: at 50 % utilisation, on-prem (~2.9 €) does not beat the European cloud (~1.8 €); at 80 % with cheap energy (~1.1 €), it beats it comfortably. The crossover sits, as the literature says, around 65–70 %. Investing in on-prem is, fundamentally, a bet that you will sustain high utilisation, and that bet is won with scheduling, not with hardware.

The uncomfortable reality: the utilisation almost nobody reaches

Here is the honest figure missing from the “on-prem is always cheaper” speeches: most inference teams in production operate at 40–65 % GPU utilisation, because of traffic variability and the limits of batching; the 80–90 % assumption that makes on-prem attractive is rarely reached outside batch-only pipelines (Spheron).

This changes the naive conclusion: if your real utilisation is 50 %, on-prem is not cheaper than the cloud, because the capex you pay for the idle GPU eats you. That is why utilisation is not a detail, it is the variable that decides the cost axis, and it connects directly with the FinOps track (the idle of A2, the chargeback of A5) and with scheduling: raising utilisation is what makes on-prem pay. A badly used cluster of your own is more expensive than the cloud; a well scheduled one is far cheaper. The cost question is not “on-prem or cloud?”, it is “can I sustain high utilisation?”.

The hidden costs of the cloud: egress

The cloud has its own small print: egress costs (taking data out of the provider). In the hyperscalers, moving data out or between regions is billed, and in AI workloads with a lot of data movement (datasets, checkpoints, embeddings) it can be a significant line item that does not appear in the GPU-hour price. The advantage of the sovereign European cloud: most of them (Lyceum among others) have adopted the zero-egress model, they do not charge for moving data out or between regions (Lyceum). When comparing, the real hyperscaler cost is GPU-hour + egress + other charges, not just the GPU-hour; ignoring it artificially inflates the hyperscaler’s competitiveness.

An example of the order of magnitude: a platform moving 50 TB/month outbound (datasets, checkpoints, responses served to systems outside the provider) at a typical egress tariff of ~0.08–0.09 €/GB pays ~4,000–4,500 €/month, that is ~50,000 €/year on egress alone, a line item the size of a third of the cost of a node of your own, invisible in the GPU-hour price. On the European cloud with zero-egress that line item is zero; on-prem, internal traffic is not billed either. That is why a fair comparison must model egress according to the real data pattern: for workloads with a lot of outbound movement, it can invert the ranking between hyperscaler and European cloud. The cloud bill is not the GPU-hour; it is the GPU-hour plus everything you move.

To this is added contract and lock-in risk: the hyperscaler’s on-demand GPU tariffs can change, commitment discounts (reserved/savings plans) tie you for 1–3 years, and migrating out, because of egress and coupling to proprietary services, has a real exit cost. On-prem and the European cloud with standard APIs (Kubernetes, S3 compatible) reduce that coupling: the same manifest and the same vLLM run on your cluster or on Scaleway without rewriting. Operational sovereignty, being able to move the workload without rebuilding it, is a value that does not appear in the tariff but weighs on a three-year decision.

The performance axis: the provider does not decide, goodput does

One point that simplifies the synthesis: performance does not depend on the provider, it depends on the hardware and the configuration. An H100 gives the same goodput on your cluster, on Scaleway or on AWS, served with the same vLLM and the same config. What decides performance is the goodput under your SLO (track B), not who hosts the GPU. So, in a comparison at equal hardware, the performance axis neutralises itself: what changes between options is cost, energy and sovereignty. The exception: if a provider gives you access to newer hardware (B200, GB200) ahead of your on-prem buying cycle, then the cloud can win on performance per GPU, a real argument in favour of the cloud for staying at the hardware frontier without capex.

The energy axis: the European and Spanish advantage

Here on-prem (or cloud) in Spain or France has a structural advantage over a hyperscaler in a dirty region. Recalling the data from the energy track:

Location	Grid carbon (gCO₂/kWh)	Price (indicative)
France (nuclear)	~20–60	low and stable
Spain (renewables + gas)	~150–170	low, volatile; solar PPA ~32.5 €/MWh
Germany	~363	high
Hyperscaler (region depends on provider)	depends; often not selectable	provider tariff

The same workload in France emits ~9× less carbon per token than in Germany, and in Spain, with a solar PPA at 32.5 €/MWh (an all-time low), the electricity cost, 30–50 % of the TCO, is low and, under contract, predictable. A sovereign cluster in Spain or France controls where the energy is consumed and with what carbon; a hyperscaler gives you the region it gives you, often with no choice of grid intensity. For CSRD reporting, that selectability is a quantifiable advantage of on-prem and the European cloud.

In concrete numbers: the example 8×H100 node (~10.4 kW × PUE 1.3 ≈ 118,000 kWh/year) emits, depending on the grid, ~2.4 t CO₂/year in France (~20 gCO₂/kWh) against ~43 t CO₂/year in Germany (~363 gCO₂/kWh), the same machine, the same work, ~18× the difference in reportable footprint just by choosing the location. That decision, which a hyperscaler in an imposed region does not let you take, is exactly what on-prem and the sovereign European cloud put in your hands. The energy axis is not an environmental detail: it is cost (the price of the kWh), compliance (CSRD) and sovereignty (control of location) all at once.

The sovereignty axis: the one that does not depend on utilisation

And here is the axis that invalidates the cheapest option if the data is sensitive. US hyperscalers are subject to the US CLOUD Act: US authorities can demand data held by a US company even if it sits in a European datacenter. For data subject to the GDPR, that is a compliance risk. Sovereign European clouds operate under EU/EFTA jurisdiction, providing data residency and GDPR compliance, and are exempt from the US CLOUD Act (Lyceum · sovereign providers). On-prem of your own is the maximum degree of sovereignty: the data does not leave your cluster.

The key difference from the other axes: sovereignty does not depend on utilisation or on volume. However much a hyperscaler cheapens the GPU-hour, for GDPR data it is not an option, the jurisdiction risk is not offset by price. It links with the ENS × ISO 42001 × EU AI Act controls and the EU AI Act mapping: compliance is a hard constraint, not an axis to optimise.

The four instruments that turn sovereignty into a concrete constraint rather than a slogan:

Instrument	What it requires	Implication for the architecture
US CLOUD Act	gives the US access to data held by US companies, wherever it is	a US hyperscaler does not guarantee jurisdictional residency even if the datacenter is in the EU
GDPR	residency and processing of personal data under EU law	requires an EU/EFTA provider or your own iron for personal data
EU AI Act	traceability, risk management and records for AI systems	favours the full control of the stack (logs, datasets, models) that on-prem gives
CSRD	verifiable reporting of environmental footprint	the selectable energy (clean grid, PPA) of on-prem and the European cloud is auditable

The operational conclusion: for a European entity processing personal data or deploying high-risk AI, three of the four technical axes can favour the hyperscaler and it can still lose, because the fourth axis, sovereignty, acts as a prior filter. That is why S2 orders the decision like this: first the sovereignty filter (which rules out the hyperscaler for GDPR data), then the optimisation of cost, performance and energy among the options that pass the filter (sovereign on-prem and European cloud).

The scorecard: the three options scored

Crossing the four axes for the three realistic options of a European platform (order-of-magnitude, illustrative figures):

Option	Cost/1M tok	Break-even	Energy/carbon	Sovereignty
Sovereign on-prem (ES/FR)	~1.1 € (high util.) / ~3 € (low)	>65–70 % util.	controllable (clean grid, PPA)	total (EU)
Sovereign European cloud	~1.5–2.2 €	no capex, pay per use	EU, zero-egress	high (EU)
Hyperscaler (US)	~2–3.5 € + egress	no capex	imposed region	not EU (CLOUD Act)

The reading of the scorecard: for data subject to the GDPR, the US hyperscaler is ruled out by the sovereignty axis, however competitive its tariff. The real decision comes down to sovereign on-prem vs sovereign European cloud, and there it is decided by utilisation and volume.

When each option wins

The honest recommendation, by scenario:

Scenario	Winning option	Why
High, sustained volume (util. >65–70 %), GDPR data	Sovereign on-prem	lowest cost/token + total sovereignty
Variable or growing volume, GDPR data	Sovereign European cloud	sovereignty without capex/idle risk
Low volume or sporadic peak	European cloud (per use)	you do not amortise the capex
No sovereignty requirement, hardware frontier	Hyperscaler	access to new hardware without capex
Hybrid (base + peak)	On-prem + European cloud (burst)	cheap base of your own, sovereign elastic peak

On-prem makes sense when there is very high, predictable utilisation (80 %+), strict sovereignty requirements, or a hyperscaler contract that works out expensive (Spheron). For sovereign platforms with a sustained base load, the winning pattern is usually the hybrid: sovereign on-prem for the high-utilisation base (where the cost per token is unbeatable) and sovereign European cloud for the peak and for growth (elastic, no capex, keeping EU jurisdiction). The best of both without giving up sovereignty.

Sizing the hybrid: how much on iron, how much on cloud

The hybrid is not “a bit of each”; it is sized with one figure: the base load percentile. The rule is to put on on-prem the load that is almost always present (the load that keeps the GPU at 75–85 %) and to send to the European cloud only the peaks that, if covered with iron, would leave GPUs idle most of the time. An example with a realistic traffic profile:

Load band	% of the time	Where to serve	Why
Base (p0–p70)	always	on-prem (1 node 8×H100 at ~80 %)	minimum cost/token, high util. guaranteed
Middle (p70–p95)	daily peak hours	on-prem if it fits, otherwise European cloud	elasticity without idle capex
Peak (p95–p100)	sporadic	sovereign European cloud (burst)	absurd to buy iron for a rare peak

With this split, the base amortises your own node at high utilisation (~1.1–1.8 €/1M tokens) and the peak is paid per use with no idle penalty (~1.5–2.2 €/1M tokens), all under EU jurisdiction. The expensive mistake is the opposite: sizing the on-prem for the peak, in which case the GPU spends most of its time idle at 30–40 %, the cost per token shoots up above 3 € and the cloud would have been cheaper. You size the iron for the base, not for the peak; the peak is exactly what the cloud does well. This principle connects with the capacity planning and the scheduling (Kueue/Volcano) of the series: the hybrid only works if the scheduler fills your own node before overflowing to the cloud.

Assumptions and sensitivity

The whole comparison hangs on assumptions that have to be declared, because moving them moves the conclusion:

Assumption	If it rises	Effect
Utilisation	50 % → 80 %	on-prem goes from losing to clearly winning
Energy price	expensive region → France/PPA	lowers on-prem TCO and carbon
Amortisation period	24 → 36 months	lowers the on-prem cost per hour
Volume	< 2M tok/day → much more	crosses the break-even towards on-prem
Egress (hyperscaler)	low → high	makes the hyperscaler dearer than the European cloud

The rule: no on-prem vs cloud comparison is valid without fixing these assumptions. One that says “on-prem is 3× cheaper” without declaring the assumed utilisation is propaganda; one that fixes utilisation, energy price, period and volume is a data point. The dossier must present the case with explicit assumptions and a sensitivity analysis, which is what makes it defensible before a committee that questions them.

Decision checklist

To take S2 from theory to decision, the questions that order the choice, in order:

Is the data subject to the GDPR, or is the system high-risk under the EU AI Act? If so, the US hyperscaler is ruled out on sovereignty; you choose between on-prem and European cloud. If not, the hyperscaler enters the cost comparison.
Can I sustain utilisation above 65–70 % on the base load? If so, on-prem wins on cost for that base. If not, the European cloud avoids paying capex for idle GPUs.
Does the traffic profile have marked peaks? If so, hybrid: base on iron, peak on European cloud. Size the iron for the base, never for the peak.
How much data do I take out of the provider each month? Model the egress; with a lot of movement, the zero-egress of the European cloud, or on-prem, win clearly.
Which electricity grid and at what price? France or Spain with a PPA lower TCO and carbon; include it in the model and in the CSRD report.
Have I fixed utilisation, energy, period and volume in writing? Without those four declared assumptions, the number is not defensible.

Whoever answers these six questions with data, not with intuition, has the case built. The series' recommendation for a sovereign European platform with a sustained base load is stable: sovereign on-prem for the high-utilisation base plus sovereign European cloud for the peak, with the hyperscaler reserved only for workloads with no sovereignty requirement where frontier hardware is needed without capex.

Limits and traps (data-driven)

Unrealistic assumed utilisation. The 80–90 % that makes on-prem win rarely happens in production (40–65 % typical). Model your real utilisation, not the ideal one.
Comparing only the GPU-hour. TCO includes energy, operation, cooling, egress (cloud) and capex (on-prem). Compare totals with the same assumptions.
Ignoring sovereignty. For GDPR data, the sovereignty axis rules out the hyperscaler before cost does; it is not negotiable with price.
Forgetting the hybrid. It is not “all on-prem or all cloud”; the base-plus-peak pattern usually dominates.
Data in USD. US comparisons are in dollars and with dirty regions; convert them to euros and to your region’s grid (Spain/France) for your case.

The synthesis of S2, in one sentence: for sovereign European data, the decision is not on-prem vs cloud in the abstract, but sovereign on-prem (high utilisation) plus sovereign European cloud (peak) against a hyperscaler that the sovereignty axis rules out, and utilisation is the variable that splits the base between the first two. The rest of the series gives the numbers for each axis; this one crosses them into the recommendation. The next synthesis article (S3) sizes the investment; this one decides the architecture.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Lenovo Press · On-Premise vs Cloud: Generative AI TCO (2026) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Lyceum · EU Sovereign Inference Platform Comparison (2026) — https://lyceum.technology/magazine/eu-sovereign-inference-platform-comparison/
Lyceum · Sovereign Cloud Providers 2026 — https://lyceum.technology/magazine/sovereign-cloud-providers-2026/
Scaleway · H100 GPU instance (precio €, soberanía UE) — https://www.scaleway.com/en/h100/
Nerd Level Tech · GPU Cloud TCO 2026: hidden fees, egress costs — https://nerdleveltech.com/gpu-cloud-comparison-2026-the-real-cost-of-ai-compute

From GPU-hour to cost per token: the metric that compares on-prem and cloud

Sun, 14 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Fourth article in the FinOps track (A4), and the one that closes out the metric that gives the whole pillar its meaning: cost per token. In A2 we saw how OpenCost assigns the cost of the iron (the pod, in €/hour); in A3, which tool to use. But the cost of the pod is not the metric that compares on-prem with cloud, nor the one the business understands: that one is cost per token (or per request). This article explains how it is really calculated (spoiler: for your own inference it is not a list price, it is €/GPU-hour divided by throughput), how to instrument it with a gateway (LiteLLM), how to attribute it by team and product, and how to compare it against European cloud. No recommendations; only the mechanics and the numbers.

The identity: from GPU-hour to token

Cost per million tokens (CPM) is the cost of the iron divided by what it produces:

$$\text{cost per token} = \frac{\text{cost of the iron (€/h)}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

This is the central formula of the FinOps track, and it has a consequence many people never quite internalise: for a self-hosted model, the cost of a token does not exist as a fixed price. It depends on two things that change: the cost of the iron (from OpenCost, in €/hour) and the throughput (from your benchmark, in tok/s). The same model on the same node costs half as much per token if you double the throughput with an optimisation, because the denominator changes, not the numerator. The cost per token of your own inference is a function of your efficiency, not a tariff.

That sets it radically apart from an external API, where the price per token really is a fixed tariff the provider charges you. Comparing the two requires turning your variable cost into a number, and that is where the iron (A2) meets the tokens (this article).

The two halves of cost per token

Cost per token comes from joining two measurements that live in different places:

The iron half comes from OpenCost (article A2): the by-usage cost of the vLLM pod, in €/hour. The token half comes from the gateway, which intercepts every request and counts tokens per model and per team. Divide, and out comes the cost per token and, crucially, per team, which is what enables chargeback.

The gateway: LiteLLM in depth

The gateway sits between the application and the engine (vLLM) or the provider, and intercepts every request. LiteLLM is the OSS reference: it proxies over 100 models and tracks spend by keys, users and teams, logging spend for all known models automatically (LiteLLM · Spend Tracking).

Its key mechanisms:

Mechanism	What it does
`response_cost`	the cost of each response, available in logging (`kwargs["response_cost"]`)
Spend logs	one row per request with tokens, model, user, key and cost (SQLite/PostgreSQL)
Tags	tag each request by team, project, environment and model tier
Budgets	budgets and rate limits per team or user
Custom callbacks	bespoke loggers for key/user/model/tokens/cost

With the spend logs table (via the database adapter) you have the detail of every request to build cost dashboards (LiteLLM · Logging), and you can tag each request with team/project/environment for per-team analysis, with notifications (Slack) and hard caps for keys that must not exceed a limit. In practice it is the platform’s token meter and cost allocator.

The self-hosted trap: LiteLLM does not know what your own token costs you

Here is the point almost nobody wires up correctly. LiteLLM tracks cost using list prices for known models (the API ones: OpenAI, Anthropic and so on). But for a model self-hosted on your vLLM there is no list price, because you pay nobody per token. If you tell it nothing, LiteLLM does not know the real cost of your own token. The solution is custom pricing: declaring to LiteLLM the cost per token of your self-hosted model (LiteLLM · Custom Pricing).

And where does that number come from? From the identity above: the cost of the iron (OpenCost) divided by the throughput (your benchmark). That is, the correct flow is:

OpenCost gives you the cost of the vLLM pod: ~11 €/hour (A2).
Your benchmark gives you the throughput: ~2,800 tok/s (track B).
The identity gives the cost per token: 11 ÷ (2,800 × 3,600 / 10⁶) ≈ 1.09 €/1M tok.
That 1.09 €/1M goes into LiteLLM as custom pricing.

Only then does the response_cost of each request reflect the real cost of your inference, rather than a zero or somebody else’s list price. Without this step, your LiteLLM cost dashboards lie about your own models. It is the explicit junction of the three tracks: iron (FinOps), throughput (benchmarking) and measurement (gateway).

Configuring custom pricing: an example

In LiteLLM, the real cost of a self-hosted model is declared in its configuration. An example with the cost per token derived from the identity (1.09 €/1M ÷ 10⁶ = 0.00000109 €/token):

# LiteLLM config.yaml — real cost of a self-hosted model
model_list:
 - model_name: llama-3.1-70b-onprem
 litellm_params:
 model: openai/llama-3.1-70b  # your vLLM endpoint
 api_base: http://vllm:8000/v1
 input_cost_per_token: 0.00000109 # €/token (from OpenCost ÷ throughput)
 output_cost_per_token: 0.00000109 # refine if you separate prefill/decode

With that, every response_cost LiteLLM records reflects the real cost of your iron. The operational note: this number must be updated when the cost of the node changes (new depreciation, a different energy price) or the throughput does (an engine optimisation). It is not a constant; it is the output of OpenCost and of the benchmark, and it is recalculated whenever either of the two changes. Automating that recalculation, from node cost and measured throughput to custom pricing, is what keeps cost dashboards honest.

Cost per token changes with throughput: read it off the sweep

Because throughput is the denominator, cost per token is not a point, it is a curve over the concurrency sweep. On the ~11 €/hour node:

Concurrency	Throughput (tok/s)	Cost per 1M tokens
1	350	~8.73 €
8	2,100	~1.46 €
16	3,330 (goodput)	~0.92 €
24	3,900 raw / 2,420 goodput	~0.78 € raw / ~1.26 € on goodput

Important readings: at low concurrency the cost per token is sky-high (the fixed cost of the node is shared among few tokens); cost falls as occupancy rises. But watch the last row: if you compute cost on raw throughput (3,900) you get 0.78 €, whereas if you compute it on goodput (2,420, the part that meets the SLO) you get 1.26 €, and the latter is the real cost, because requests that violate the SLO do not count as useful production. An honest cost per token is computed on goodput, not on catalogue throughput. This is where the cost track meets the benchmarking one: the correct denominator is goodput.

Cost per request: not only per token

Cost per token is the unit; cost per request is what a complete request bills. A request consumes input tokens (prompt) plus output tokens (generation), and both count:

$$\text{cost per request} = (\text{input tokens} + \text{output tokens}) \times \text{cost per token}$$

With nuances: in many engines the prefill (input) and the decode (output) have different costs per token (decode is memory-bound and more expensive per token), so a fine-grained cost model distinguishes the two. For most cases a weighted average cost per token will do; to optimise, it pays to separate prefill and decode. LiteLLM records input and output tokens separately in its spend logs, so the data is there to refine with.

Cost per request is what connects to the product: if a feature consumes 1,500 tokens on average (1,200 of prompt plus 300 of output) and the cost is 1.09 €/1M, each use costs ~0.0016 €, the basis for setting margin, as seen in the FinOps introduction.

Cost per model and per tier

A platform does not serve a single model. Cost per token varies by model and by configuration, and the gateway tells them apart. An 8B and a 70B on the same node have very different costs per token: the 8B yields far more throughput (more tokens/s) for the same GPU-hour, so its cost per token is a fraction of the 70B’s. The design consequence: routing each request to the cheapest model that meets the quality bar is a direct cost lever, since serving a 70B where an 8B suffices means overpaying per token.

Model (example, same node)	Relative throughput	Relative cost per token
8B	high	low
70B FP8	medium	medium
70B FP16	low	high

The gateway, with its per-model custom pricing, makes this difference visible and enables cost-based routing: classify the request and send it to the appropriate tier. It connects with the L7 router of the serving track: the gateway not only measures cost, it can decide on the basis of it. And quantisation (FP16→FP8) shows up again as a lever: the same model in FP8 raises throughput and lowers cost per token, moving the figure without changing model.

Attribution: from cost to team and product

The reason for putting a gateway in place is not only to measure total cost, it is to share it out. By tagging each request with team, product and environment, LiteLLM builds the split by business dimension:

With per-team budgets and hard caps per key, the gateway does not only report: it governs. A team that goes over budget gets an alert; a key with a hard cap cannot spend any more. It is the FinOps Operate phase applied to the token.

A combined report example

Joining the iron (OpenCost) and the tokens (gateway), the monthly cost-per-token-per-team report on the example node:

Team	Tokens/month	Applied cost per token	Charged cost
A · chat-prod	240M	1.09 €/1M (70B FP8)	~262 €
B · batch	90M	0.40 €/1M (8B)	~36 €
C · experimentation	15M	1.09 €/1M (70B)	~16 € + idle

What it reveals: A consumes a lot but at an efficient cost (FP8, high utilisation); B uses a cheap model (8B) for its workload; C consumes little but, as we saw in A2, drags along the idle cost of its underused GPU. The token report plus the OpenCost iron report give the full picture: who spends, on which model, with what efficiency. It is the chargeback that neither half would produce alone. And the per-team figure is what gets taken into the budget conversation: not “the cluster costs X”, but “your team consumed Y million tokens at Z €/million”.

Governing spend: budgets, caps and alerts

Measuring without governing is a dashboard; the value lies in closing the loop. LiteLLM allows three levels of control over spend:

Control	What it does	When
Soft budget	alert (Slack) when a threshold is crossed	tracking teams
Rate limit	limits requests/tokens per minute	avoiding cost spikes
Hard cap	the key cannot spend any more	keys that must never overrun

The healthy pattern: soft budgets with alerts for the teams (so they adjust their consumption), rate limits to smooth spikes, and hard caps only on critical keys (a service that must not run away). With this, cost per token stops being an after-the-fact figure and becomes a real-time control: the FinOps Operate phase, applied to inference.

On-prem vs European cloud comparison (the reason for all of it)

Cost per token exists to answer one question: is it cheaper to serve on my own iron or to rent? With the identity and European figures:

Option	Cost per 1M tokens	Notes
Depreciated on-prem (8×H100, 2,800 tok/s)	~1.09 €	your iron, your sovereignty
European cloud Scaleway (8×H100, on-demand)	~2.17 €	EU, no iron to operate
External API (proprietary model)	provider tariff	no data sovereignty

Cost per token is what makes the incomparable comparable: your variable cost (iron ÷ throughput) against a provider’s fixed tariff. And it exposes the lever: if an optimisation raises your throughput from 2,800 to 4,200 tok/s, your on-prem cost drops to ~0.73 €/1M, widening the advantage over renting. That is why benchmarking (track B) is, indirectly, the tool that moves cost per token the most: every goodput improvement translates into this number.

From cost per token to product price

Cost per token closes the loop with the business when it becomes cost per unit of product. The chain: cost per token (from the identity) × tokens per use of a feature = marginal cost of that feature. With that, three business decisions become data-backed:

Setting a price. If serving a feature costs 0.0016 € per use, the price to the customer has to cover it with margin; cost per token is the floor of pricing.
Spotting loss-making features. A feature that consumes many tokens (long prompts, lots of output) may cost more than it contributes; cost per request reveals it before you scale.
Comparing build vs buy at product level. On-prem cost per token against an API tariff, for that specific workload, decides whether to serve in house or consume externally.

This is the ultimate reason for the whole FinOps track: it is not infrastructure accounting, it is the quantitative basis for product and architecture decisions. A measured and attributed cost per token turns “is this profitable?” from an intuition into a calculation.

Detailed comparison: on-prem, European cloud and API

Extending the comparison with scenarios, for the same job (1M tokens):

Option	Cost/1M tok	Sovereignty	Where it fits
Depreciated on-prem, high utilisation	~0.7–1.1 €	full (EU)	high, sustained volume
Depreciated on-prem, low utilisation	~2–4 €	full (EU)	bad: idle drives cost up
European cloud (Scaleway, on-demand)	~2.2 €	EU	variable volume, no iron to operate
European cloud (reserved/committed)	~1.5–1.8 €	EU	medium committed volume
External proprietary API	provider tariff	non-EU	prototype / no sovereignty requirement

The table shows two things. First, on-prem only wins if utilisation is high, since at low occupancy idle makes it dearer than cloud, which connects cost per token with scheduling and utilisation (the idle of A2). Second, the external API, however competitive its tariff, loses data sovereignty, an axis that for GDPR data is not negotiable. Cost per token is the metric that puts the first three on the same scale; sovereignty is the constraint that rules out the last one for certain kinds of data.

Prompt caching and effective cost

One nuance refines the real cost per token: prefix caching. When many requests share the same prefix (a long system prompt, a repeated context), the engine caches its prefill compute and does not recompute it on every request. That means cached input tokens cost almost zero the second time and thereafter, so the effective cost per token of a workload with a high cache hit rate is lower than the nominal cost. External APIs already reflect this with a reduced price for cached tokens; in self-hosted the effect is that throughput rises (less prefill to recompute), which by the identity lowers cost per token.

The implication for the cost model: if your workload has shared prefixes (RAG over the same corpus, agents with the same system prompt), the real cost per token is lower than the naive calculation gives, and it pays to measure the prefix caching hit rate so as not to overestimate cost. And the reverse: a workload of unique, never-repeated prompts gains nothing from the cache, and its cost per token is the full nominal figure. Cost per token, once again, is not a constant: it depends on the model, the precision, the utilisation, the goodput and the reuse pattern of the workload. Measuring it on your real traffic, not on a synthetic benchmark without a cache, is what gives the number you actually pay.

State of the art 2026

The gateway as standard: LiteLLM (and alternatives) are consolidating as the layer that measures tokens, attributes cost and governs budgets in the LLM stack.
FOCUS for tokens: layers on top of the gateway generate FOCUS-compatible logs, integrating AI cost with the rest of cloud spend (FinOps introduction).
Custom pricing as a bridge: the OpenCost→gateway connection via custom pricing is the standard way to give self-hosted models a real cost.
Cost per token as a business KPI: increasingly, cost per token (and per product) is the metric the business demands, not the cost of the pod.

Limits and traps (data-driven)

Forgotten custom pricing. Without declaring to LiteLLM the real cost of your self-hosted token, the response_cost of your own models is zero or someone else’s price. Close the OpenCost→gateway bridge.
Static throughput. Cost per token changes with throughput; if you optimise the engine but do not update the custom pricing, the reported cost drifts out of date.
Forgetting prefill/decode. An average cost per token is fine for reporting; to optimise, separate input and output.
Measuring without governing. Spend logs without budgets or caps make a pretty dashboard; the value lies in closing the loop with alerts and caps.
Comparing tariffs with costs without matching assumptions. Your 1.09 €/1M assumes a utilisation and a throughput; the provider’s tariff does not. Fix the assumptions before comparing.

With cost per token solved, the FinOps track has its central metric; the following articles (A5 multi-tenant chargeback, A8 TCO model) build on it. The full chain, iron (OpenCost) ÷ throughput (benchmark) = cost per token, measured and attributed by the gateway, is what turns the cost of AI into a defensible number, in euros and per team.

Closing

Cost per token is the metric that reconciles the three tracks of the series into a single number, and its underlying lesson is counter-intuitive: for your own inference, the cost of a token is not a price, it is the reflection of your efficiency. It comes from dividing the cost of the iron (OpenCost) by useful throughput (the goodput of your benchmark), and it becomes visible and governable with a gateway (LiteLLM) that counts the tokens, attributes them by team and applies budgets. The mistake that invalidates the whole exercise is leaving custom pricing at zero and believing your own inference is “free”; the right move is to close the OpenCost→gateway bridge and keep the number up to date when the node cost or the throughput changes. Done that way, cost per token answers with data the one question the business keeps repeating, is it worth serving this on my own iron, and it does so in euros, per team and per product, comparable against European cloud and against any API. For a sovereign architecture proposal, it is the column of the dashboard that translates all the cost engineering into the sentence that closes a budget meeting: this is the cost per token, this is how it is measured, and here is the bench to reproduce it.

Sources

LiteLLM · Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM · Custom Pricing (modelos self-hosted) — https://docs.litellm.ai/docs/proxy/custom_pricing
LiteLLM · Logging — https://docs.litellm.ai/docs/proxy/logging
LiteLLM · documentación general — https://docs.litellm.ai/docs/
Statsig · LiteLLM cost tracking — https://www.statsig.com/perspectives/litellm-cost-tracking
StackPulsar · LiteLLM production monitoring (2026) — https://stackpulsar.com/blog/litellm-production-monitoring/

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

Energy benchmarking in AI: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The energy and carbon references are European (France, Germany, Spain), since this is a sovereign proposal. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Fourth article in the data series and deep dive on the energy axis. Measuring the watts of an AI workload looks like a sustainability detail, but it is at once a cost lever (electricity is 30–50 % of TCO), an axis of sovereignty (carbon per token depends on the country) and, increasingly, a regulatory obligation (emissions reporting). This article inventories the metrics, where each figure physically comes from, the data sheet for each measurement framework, and how to go from watt to carbon with European figures. No recommendations; only data and methodology, because in energy, even more than in performance, the measurement method changes the result.

Why measuring energy matters (with numbers)

Three quantified reasons, connecting with the opening article:

Reason	Figure	Implication
Cost	electricity = 30–50 % of TCO	measuring energy is measuring almost half the cost
Sovereignty/carbon	France ~9× fewer gCO₂/token than Germany	the country of the cluster is a lever
Scale/regulation	datacenters ~460 TWh in 2025 (≈1.8 % worldwide), >800 TWh projected for 2028	emissions reporting (CSRD) mandatory

At 700 W per H100, an 8-card node draws of the order of 5.6 kW from the boards alone, ~49,000 kWh a year. Applying PUE 1.4 that becomes ~68,700 kWh/year, and there the country decides the bill: at the French wholesale price (~0.058 €/kWh) that is ~3,980 €/year of electricity per node; in Germany (~0.091 €/kWh), ~6,250 €/year, 57 % more for the same iron and the same work. Multiplied across a fleet, it is a first-order cost line, and a carbon footprint that, in Europe, has to be measured and reported. Energy stopped being an engineering detail and became a business and compliance variable.

The energy metrics

Metric	Definition	Unit
Mean power	average instantaneous consumption	W
Energy per token	energy consumed per generated token	J/token
Energy per inference	energy per request	µJ – Wh
Total energy	integral of power over time	Wh, kWh
PUE (Power Usage Effectiveness)	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The two base identities of the whole axis:

$$\text{energy per token (J)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)}}$$ $$\text{carbon} = \text{energy (kWh)} \times \text{PUE} \times \text{grid intensity (gCO}_2\text{/kWh)}$$

The first has the same shape as cost per token: it shares the throughput denominator, so raising tokens/s lowers energy per token. The second brings in the two multipliers external to the compute, the datacenter PUE and the country’s grid intensity, which can change carbon per token by an order of magnitude without touching the stack.

Worked example: energy per token of an 8×H100 node

To anchor the identities with numbers, the end-to-end calculation on the example node (8×H100, Llama 3.1 70B FP16, vLLM at 2,800 tok/s):

Step	Calculation	Result
Board power	8 × 700 W	5,600 W
Energy per token (boards)	5,600 ÷ 2,800	2.0 J/token
Energy per token (with PUE 1.4)	2.0 × 1.4	2.8 J/token
Energy per 1M tokens (boards)	2 × 10⁶ J	0.56 kWh
Energy per 1M tokens (with PUE)	0.56 × 1.4	0.78 kWh

From there come the electricity cost and the carbon, by country:

Country	Electricity cost / 1M tok	Carbon / 1M tok
France (0.058 €/kWh; ~40 gCO₂/kWh)	~0.045 €	~31 gCO₂
Spain (0.054 €/kWh; ~160 gCO₂/kWh)	~0.042 €	~125 gCO₂
Germany (0.091 €/kWh; ~363 gCO₂/kWh)	~0.071 €	~283 gCO₂

And the link with performance: if an optimisation (FP8, say) raised throughput to 4,200 tok/s, energy per token would drop to ~1.33 J/token (boards) and everything below it in the same proportion. Electricity cost and carbon per token fall with throughput, by the identity. Note: these are board + PUE figures; the complete node (CPU, NICs, PSUs) consumes somewhat more, and energy measured with a power meter can differ from energy estimated with nvidia-smi.

Where the figure comes from: the measurement layers

Not every energy figure is equally reliable. The figure can come from four layers, with increasing accuracy and decreasing context:

The GPU is the dominant source in LLM inference, and its power is read from a hardware counter (NVML/nvidia-smi, aggregated by DCGM, the same basis as GPU observability). RAPL gives CPU and DRAM. The eBPF + model layer (Kepler) splits the total watt between containers and pods without instrumenting the application. And the physical power meter is the ground truth against which estimates are calibrated: no software estimate is better than the meter, only more convenient.

How it is instrumented (and where the error creeps in)

Four instrumentation decisions explain most of the discrepancies between energy measurements:

Sampling rate. nvidia-smi/DCGM read power at intervals; sampling that is too coarse misses the peaks and underestimates the energy. It must be fine enough to capture the power variation between prefill (high) and decode (lower).
Board vs node. GPU (board) power does not include CPU, NICs, fans or PSU losses (typically an extra 10–20 %). Comparing a board energy figure with a complete-node one is comparing different things.
Idle baseline. A powered GPU with no work already draws power. Is that idle attributed to the workload or discounted? The decision changes energy per token, and idle matters: it is real energy that someone pays for.
Time window. Energy is the integral of power over time; the measurement window has to line up exactly with the workload (without counting warm-up or shutdown), or the number does not correspond to the work measured.

These decisions, as in performance benchmarking, mean that the measurement method matters as much as the system measured. A J/token without a stated sampling rate, board/node boundary and idle treatment is not comparable.

Frameworks, tool by tool

Zeus (ml.energy, University of Michigan)

What it does: measure and optimise the energy of deep learning workloads. Method: NVML counters. Scope: NVIDIA and AMD GPUs, CPU, DRAM, Apple Silicon and NVIDIA Jetson. Licence: OSS; academic backing (NSDI'23 paper). Differentiator: it does not only measure, it also optimises (power capping, frequency selection) to cut energy without significant loss of performance (Zeus Project, NSDI'23). It is the reference tool when the goal is to lower J/token, not just to measure them.

CodeCarbon

What it does: estimates the energy and the CO₂eq of compute processes. Method: reads the GPU with nvidia-smi and estimates CPU/RAM. Scope: CPU + GPU + RAM. Licence: OSS (Python). Differentiator: it integrates into ML flows and CI pipelines, and it is the one that comes closest to the power meter among the Python tools (comparison). It brings carbon context out of the box. Limit: a high-level estimate, less exact than direct hardware measurement.

CarbonTracker

What it does: tracks energy and carbon at GPU run time. Scope: GPU only, it measures neither CPU nor memory. Licence: OSS. Differentiator: lightweight, easy to add to a training/inference script. Limit: partial coverage (no CPU/DRAM), so it undercounts the total system energy.

Scaphandre

What it does: a power metrology agent written in Rust. Method: RAPL (reads /proc, splits joules by CPU ticks). Scope: process or VM, with the raw accuracy of the hardware layer. Licence: OSS. Differentiator: the highest direct accuracy among the general-purpose tools, at the cost of bringing no carbon context (it is pure metrology) (Scaphandre · GitHub). Limit: focused on CPU/RAPL; for GPU it has to be combined with NVML.

Kepler (CNCF)

What it does: a Prometheus exporter that measures energy at container, pod and node level in Kubernetes. Method: eBPF + ML models to estimate consumption per workload. Language: Go + C. Licence: Apache 2.0 (the sustainable-computing-io project) (Kepler · GitHub). Differentiator: it is the cloud-native option for attributing energy per pod without instrumenting the app, just as OpenCost attributes cost. Limit: attribution via eBPF + model is estimation, not direct measurement; its accuracy depends on the model.

MLPerf Power (MLCommons)

What it does: a standardised benchmark of energy efficiency, from µW to MW. Maintainer: MLCommons. Differentiator: cross-vendor comparability of efficiency, with a dataset spanning several years and workload versions (MLPerf Power, arXiv 2410.12032). It is to energy what MLPerf Inference is to performance: the comparison standard between vendors.

Comparison table

Tool	Method	Scope	Licence	Accuracy / note
Zeus	NVML / counters	NV+AMD GPU, CPU, DRAM, Apple, Jetson	OSS	measures and optimises DL energy
CodeCarbon	nvidia-smi + estimation	CPU+GPU+RAM, CO₂eq	OSS (Python)	the closest to the power meter among Python tools
CarbonTracker	GPU runtime	GPU only	OSS	lightweight; no CPU/mem
Scaphandre	RAPL (/proc)	process/VM	OSS (Rust)	raw accuracy; no carbon context
Kepler	eBPF + model	container/pod/node (K8s)	Apache 2.0 (CNCF)	cloud-native; model-based estimation
MLPerf Power	standardised measurement	µW to MW	MLCommons	cross-vendor comparability

Accuracy: who to believe

The difference between tools is not a nuance: Scaphandre offers the raw accuracy of the hardware layer (RAPL), while the Python ones such as CodeCarbon give higher-level, carbon-aware estimates, with more context but less direct exactness; among the Python tools, CodeCarbon is the one that comes closest to the power meter, followed by CarbonTracker, with variability across infrastructures (comparison). The rule: for sizing and design, a software estimate will do; to defend a figure before a committee or an auditor, calibrate against a power meter at least once, and use the software tool as a continuous proxy. The measured figure beats the estimated one; the rest is convenience.

Energy leaderboards and benchmarks

To compare efficiency between models without building your own test bench:

Resource	What it provides
MLPerf Power	standardised efficiency measurement, from µW to MW
HF AI Energy Score	comparable per-model energy efficiency ratings (HF)
ML.ENERGY Benchmark	automated measurement and optimisation of inference energy
awesome-green-ai	curated index of Green AI resources and tools (repo)

These resources give the order of magnitude of the energy per token of each model, useful for choosing a model on efficiency at the design stage. But they do not replace measurement on your hardware and your workload: a leaderboard measures on one specific bench, not on your node.

From watt to carbon: France, Germany, Spain

Here the energy axis turns into a sovereignty argument. Carbon per token comes from multiplying the energy by the country’s grid intensity, which in Europe varies by almost two orders of magnitude:

Country	Wholesale (€/MWh, Mar 2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: wholesale prices (TradingEconomics), Germany’s carbon ~363 gCO₂/kWh (2024 average, Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). Hourly intensity comes from ElectricityMaps.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂, a factor of ~9×, without touching a line of the stack. Placing the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever (EU AI Act, controls ENS × 42001 × EU AI Act).

Energy efficiency levers

Once it is measured, it can be brought down. The levers, ordered by the identity J/token = W ÷ throughput (lower the numerator or raise the denominator):

Lever	Mechanism	Effect
Quantisation (FP8/INT4)	more tokens per GPU-hour, less VRAM/KV	raises throughput → lowers J/token
Batching / continuous batching	amortises the fixed power cost	raises throughput → lowers J/token
Power capping (Zeus)	limits GPU power	lowers W with little loss of throughput
Efficiency-aware scheduling	consolidate workloads, power down idle GPUs	eliminates idle consumption
Location (country)	cleaner grid	lowers carbon per token
Avoiding over-reasoning	fewer reasoning tokens that add nothing	less compute → less energy

Quantisation (see quantisation for inference) is the highest-return lever because it moves all three axes at once: it raises throughput, lowers cost and lowers energy per token. Zeus’s power capping is energy-specific: it trims peak power with little performance loss, lowering the W in the numerator. And idle, the GPU powered but not working, is pure wasted energy that scheduling recovers.

From design to production: where energy is measured

As with the other axes, energy is measured at three moments, with different tools and increasing accuracy:

Moment	Tool	What it gives
Design	TDP × hours (datasheet)	ceiling estimate for sizing
Benchmark	Zeus, MLPerf Power, power meter	energy per token measured on a controlled bench
Production	Kepler (eBPF), DCGM	continuous per-pod consumption in the cluster

The design figure (TDP) serves to size the utility feed and the cooling, but it overestimates (the GPU is rarely at TDP 100 % of the time). The benchmark figure is the one used for the proposal’s cost and carbon per token, because it is measured and reproducible. And the production figure (Kepler/DCGM) closes the loop: it checks that real consumption matches the budget and feeds the footprint report. The three have to be traceable to each other, which is what the harness in article S4 does.

Energy in TCO and European regulation

The closing of the axis: energy is not only sustainability, it is cost and compliance.

Cost: being 30–50 % of TCO, every J/token improvement translates directly into euros. In the example, moving from Germany (~0.091 €/kWh) to France (~0.058 €/kWh) lowers the electricity share by ~37 % on top of the carbon.
Regulation (CSRD): European companies must report their carbon footprint (scope 2). A low-carbon grid (France) directly improves that reported figure, and measuring energy per token stops being optional and becomes a reporting requirement.
EU AI Act: efficiency and energy consumption form part of the documentation expected of AI systems, which makes energy measurement part of the technical file.

One reporting nuance worth knowing: the electricity footprint is scope 2 (indirect emissions from purchased energy), and it can be accounted for by location (the real intensity of the country’s grid) or by market (according to the guarantees of origin contracted). Location-based accounting is the one that genuinely rewards putting the cluster on a clean grid such as the French one; market-based accounting allows green energy to be “bought” through certificates. Knowing which one your CSRD report requires changes which lever (physical location vs contract) lowers the figure, and both start from measuring energy per token, which is what this axis provides.

In other words: measuring energy covers at once a first-order cost line, the carbon and sovereignty argument, and a regulatory obligation. For a sovereign European proposal, the energy axis is not the “nice sustainability bit”: it is one of the three columns with a number attached.

Checklist for a reproducible energy measurement

For an energy or carbon figure to be defensible before a committee or an auditor, it has to come with its context. The minimum to fix and publish:

What to fix	Why
Tool + version	each layer (RAPL, NVML, eBPF) and tool measures differently
Sampling rate	coarse sampling misses peaks
Board vs node boundary	±10–20 % depending on whether CPU/PSUs are included
PUE used	it multiplies all the compute energy
Source and time of the grid intensity	carbon varies ~9× by country and by hour
Idle treatment	attributed or discounted changes the J/token
Model, precision and workload	energy depends on what is served and how
Calibration vs power meter	at least once, to validate the estimate

The rule: if you cannot hand over this table alongside the gCO₂/token figure, the figure is not auditable. The reproducible harness of article S4 records all these parameters so that the energy figure is as traceable as the cost and performance ones, and so that the reported footprint (CSRD) holds up under audit.

State of the art 2026

LLM-specific resolution is missing: general-purpose tools measure well at machine or process level, but they lack LLM-specific resolution (energy per token, per prefill/decode phase), and the simulators lack emissions modelling, an open gap in the field.
eBPF as a trend (Kepler) for attributing energy per pod without instrumenting the app, in parallel to how OpenCost attributes cost.
Standardisation of comparison via MLPerf Power and HF AI Energy Score.
Optimisation, not only measurement (Zeus): the field is moving from “how much do I spend” to “how do I spend less without losing performance”.

Limits and traps (data-driven)

Estimation vs measurement. An nvidia-smi figure is not a power meter. Calibrate against hardware at least once before defending a number.
Partial coverage. CarbonTracker measures GPU only; Scaphandre, mostly CPU/RAPL. Add the layers or you undercount the total energy.
Forgetting PUE. Board energy does not include cooling or losses; without PUE you underestimate the real consumption and carbon.
Carbon without a country. A gCO₂/token without a stated grid (and hour) means nothing: it varies ~9× between France and Germany.
Confusing energy with sustainability. Here it is cost and compliance, with a number; treating it as a “green” extra is losing one of the three columns of the proposal.

That closes the run of introductions: cost, performance and energy, each with its frameworks and its methodology. The rest of the series goes tool by tool through to the dashboard that underpins the sovereign architecture decision, with European figures, measured and reproducible.

Closing

Energy is the axis easiest to dismiss as “the green one” and the one that hides the most: it is almost half the cost, it defines the footprint Europe requires to be reported, and its carbon per token changes ~9× depending on whether the cluster sits in France or in Germany. Measuring it properly, with the right layer, the right sampling, PUE included and the country’s grid, turns a soft sustainability argument into a hard column of the proposal, with a number and a source. And the sovereign conclusion stands on its own with the data in this article: an inference platform in France or Spain is at once cheaper, cleaner and compliant with European jurisdiction than the same platform on a US hyperscaler, and that is demonstrated with measured J/token, gCO₂/kWh by country and euros, not with intentions. The energy axis does not decorate the proposal: it closes it.

Sources

Zeus Project (ml.energy, UMich) — https://ml.energy/zeus/
Zeus · USENIX NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Kepler (CNCF, eBPF) · GitHub — https://github.com/sustainable-computing-io/kepler
Scaphandre · GitHub — https://github.com/hubblo-org/scaphandre
CarbonTracker · GitHub — https://github.com/saintslab/carbontracker
MLPerf Power (arXiv 2410.12032) — https://arxiv.org/html/2410.12032v2
HF AI Energy Score — https://huggingface.github.io/AIEnergyScore/
awesome-green-ai — https://github.com/samuelrince/awesome-green-ai
TradingEconomics · precio electricidad Francia/Alemania/España — https://tradingeconomics.com/france/electricity-price
EEA · intensidad de emisiones de la electricidad en Europa — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbono ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany

Taking RAG to the CPU: separating the data plane from the generation plane

Thu, 11 Jun 2026 03:20:00 +0000

Third piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings: sharing one GPU between workloads (time-slicing, MPS, MIG) and serving several models on one GPU (swap + sleep) attack the split inside the GPU. This one attacks the split outside: which parts of RAG never need to touch the GPU at all. The closing piece of the series, the end-to-end sovereign assistant (fourth instalment, in preparation), assembles the complete system where these pieces fit together.

TL;DR

A RAG system is not one thing, it is three phases with opposing compute profiles, and putting them all on the GPU “because it is AI” is an allocation mistake. (1) Build/ingest, embedding the corpus and building the index, is batch, throughput-bound work with no latency SLA: its natural home is the CPU. (2) Query-time retrieval, embedding the query, HNSW search, RRF fusion, lightweight rerank, is mostly CPU, with caveats only for heavy reranking; vector search always ran on CPU, even in stacks sold as “GPU”. (3) Generation, the LLM producing the answer, is latency-bound, and there the GPU is irreplaceable: a 7B on CPU gives a time to first token of seconds, unacceptable for chat. The technical key to why (1) and (2) fit on CPU: the embedder is not an LLM. bge-m3 is around 568M parameters (an XLM-RoBERTa encoder), not 7B+; in int8 it occupies about 580 MB and activates fast integer compute paths (Intel AVX-512 + VNNI + AMX on 4th-gen Xeon and later; NEON SDOT/UDOT on ARM). The runtimes are ready: TEI with a CPU backend (same OpenAI API /v1/embeddings and /rerank), Qdrant’s fastembed (ONNX-CPU), bge-m3 in ONNX int8 with its three heads (dense/sparse/ColBERT). The Intel + Hugging Face blog post with Optimum Intel and fastRAG reports up to around 10× on indexing for BGE-large int8 on a 4th-gen Xeon (their benchmark figure, encoding-only; I quote it and qualify it below). The operational conclusion: separate the data plane (CPU) from the generation plane (GPU). In the 4×H100 cluster, no H100 should be spent re-indexing a corpus that changes once a day. That goes to the generic CPU fleet (Xeon AMX, NUCs), and the H100s are reserved for generation and, at most, for rerank peaks or large 7B embedders. What does not go down to CPU: interactive generation, massive reranking at high QPS, re-indexing with a strict real-time SLA, and 7B embedders (gte-Qwen2, NV-Embed).

The analogy: the library and the librarian

Picture a serious research library. There are three distinct jobs, done by different people, on different clocks.

The first is cataloguing. Boxes of new books arrive; someone opens them, classifies them, assigns a shelf mark, indexes them in the catalogue and puts them on the right shelf. It is patient background work, done at night or between other tasks. Nobody is standing over you with a stopwatch waiting for today’s batch to be catalogued: what matters is that tomorrow it is done and done well. It is pure throughput: how many books you catalogue per hour, not how long you take on one specific book. This is ingestion.

The second is handling an enquiry at the desk. A reader turns up and asks about a topic. The librarian goes to the catalogue, which already exists, finds half a dozen relevant shelf marks, fetches them and puts the books on the desk. It is fast, light, and consists of searching an index that already exists, not building it. This is retrieval.

The third is writing a reasoned report from those books. The reader, or an expert you commission, reads the six books, compares them, synthesises, and writes an argued answer with citations. This is slow, demands a very well-trained mind, and the reader is waiting: here there is a human stopwatch. This is generation, the LLM.

The moral is about staff allocation. You do not put your star writer, expensive, scarce, with a queue of people waiting for reports, to catalogue boxes of books at dawn. Cataloguing is done by a large, cheap team that works through the night without rushing. The star writer touches only what genuinely needs their mind: writing. In our system, the star writer is the H100, and cataloguing at dawn is corpus ingestion. Spending the H100 on re-indexing is exactly the error of putting the writer to work labelling boxes.

The rest of the post is, essentially, which parts of the library work the cheap CPU team can do (almost all of them) and which is irreducibly the GPU writer’s job (only the last one).

The three phases and their compute profiles

The confusion that GPU over-provisioning feeds on is treating “RAG” as a monolithic block that “uses AI, therefore goes on the GPU”. No. RAG is a data pipeline with a generative model plugged in at the end. The correct architectural boundary does not separate “what uses models” from “what does not”, because both sides use models, but throughput-bound from latency-bound, which is the same as separating the data plane from the generation plane.

Why ingestion fits on CPU: the embedder is not an LLM

The whole argument rests on a size asymmetry that gets overlooked. People hear “embeddings” and “generation” and put them in the same bag of “big models that need a GPU”. But the embedding encoder and the generative LLM are two orders of magnitude apart in parameters.

bge-m3, the reference multilingual embedder, is an XLM-RoBERTa of around 568M parameters (model card, paper arXiv:2402.03216). Its sibling reranker, bge-reranker-v2-m3, is built on the same base and lands at roughly the same 568M parameters (model card). Compare that with an entry-level generative LLM: a Llama 3.1 8B has around 14× more parameters, and the big production models sit at 70B+. A 568M encoder is, in compute budget terms, a different animal.

Two structural differences make that encoder comfortable on CPU:

It is an encoder, not an autoregressive decoder. It processes the whole sequence in a single forward pass and emits the vector. There is no token-by-token decode, no growing KV cache, none of the memory-bound generation phase that kills the CPU. It is a dense matrix pass and that is it.
It quantises to int8 with almost no loss. In int8, bge-m3 occupies on the order of 580 MB and, above all, activates the integer compute paths that a modern CPU executes quickly: matrix instructions such as Intel AMX (Advanced Matrix Extensions, 4th-generation Xeon and later), AVX-512 with VNNI (Vector Neural Network Instructions) on earlier Xeons, and NEON SDOT/UDOT on ARM. The quality loss from going FP32 to int8 on these models usually stays below 1% of retrieval recall, practically invisible (Intel + Hugging Face, CPU Optimized Embeddings).

Let us quantify the int8 size. For $P = 568 \times 10^6$ parameters at 1 byte each:

$$\text{size}_{\text{int8}} \approx 568 \times 10^6 \text{ params} \times 1 \text{ byte/param} \approx 568 \text{ MB}$$

In other words, the model fits in the cache and RAM of any server or NUC without blinking, and the bottleneck is integer compute, exactly what AMX/VNNI accelerate. There is nothing in this profile that asks for a GPU.

Runtimes that already do this effortlessly

Nothing needs inventing. The CPU ecosystem for the data plane is mature:

Text Embeddings Inference (TEI) from Hugging Face: a Rust server with CPU backends via ONNX Runtime (recommended) or Intel MKL, and OpenAI-compatible endpoints (/v1/embeddings) plus /rerank (TEI repo). That is, the data plane on CPU exposes exactly the same HTTP contract as a GPU server; the rest of the system never learns which silicon is behind it.
fastembed from Qdrant: a lightweight library that loads embedders on ONNX-CPU and generates dense, sparse and ColBERT vectors (fastembed repo). Designed from the start to run without a GPU.
bge-m3 in ONNX int8 with its three heads (dense / sparse-lexical / ColBERT multivector) exported and quantised, ready for ONNX Runtime CPU.

The Intel and Hugging Face figure that anchors viability: in their benchmark with Optimum Intel + fastRAG on a 4th-generation Xeon (8480+, 56 cores, 1 socket), the int8 variant of BGE-large reaches up to around 10× indexing throughput against FP32 (HF blog, Haystack/deepset). The small print needs reading, so I read it: that ~10× is encoding-only (tokenisation excluded), at sequence length 256, comparing int8 against FP32 on the same CPU. It is not “CPU 10× faster than GPU”, it is “int8 10× faster than FP32 on CPU”. It is still the relevant figure: it tells you that with quantisation the CPU goes from unworkable to perfectly useful for batch ingestion.

Viability table: CPU for each component?

This is the operational table. The column that matters is the caveat, because a bare “yes” or “no” lies.

Component	CPU viable?	Caveat
Chunking (slicing the corpus)	Yes, always	It is regex, parsing and windows; it never had anything to do with the GPU.
Ingest embedding `bge-m3` dense	Yes, its best case	Overnight batch, int8 + AMX/VNNI. This is exactly what the CPU shines at.
Sparse / SPLADE / BM25 head	Yes, CPU-native	Lexical work is pure inverted index; the GPU adds nothing here.
Building the HNSW index (Qdrant, pgvector)	Yes, always CPU	The HNSW graph build is CPU by design in these engines.
Query embedding (online)	Yes	A single short text; tens of ms on CPU, more than enough for chat.
Dense + sparse search + RRF	Yes	Vector search always ran on CPU, even in “GPU” stacks. RRF is sorting lists.
Cross-encoder reranker `bge-reranker-v2-m3` top-20/50	Yes, with care	A cross-encoder evaluates $k$ query-doc pairs: cost $\propto k$. Over 20-50 candidates it works; over hundreds at high QPS, no.
ColBERT late interaction	Marginal on CPU	The token-by-token matrix product of late interaction is heavy; viable at low volumes, suffers with QPS.
LLM generation	No, in practice	A 7B on CPU gives TTFT in seconds. Interactive latency = GPU.

Two rows deserve underlining because they dismantle myths.

“Vector search needs a GPU.” False from the start. The HNSW index, the navigable small-world graph used by Qdrant, pgvector with vector/halfvec, Milvus in CPU mode and almost everything else, was always built and traversed on CPU. Even the stacks advertised as “GPU-accelerated RAG” do the embedding on GPU but the ANN search stays on CPU in the vast majority of deployments; the GPU variants of the index (CAGRA and similar) are the expensive exception, not the norm, and are justified only with billions of vectors and extreme QPS. For a corporate corpus of millions of chunks, HNSW on CPU resolves in single-digit milliseconds.

“The reranker is a model, therefore GPU.” The bge-reranker-v2-m3 reranker is a cross-encoder of around 568M: it runs on CPU. The caveat is the number of pairs. A cross-encoder does not produce a reusable vector; it evaluates the (query, document) pair jointly, so its cost grows linearly with the number of candidates $k$:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking the top-20 or top-50 coming out of hybrid retrieval is perfectly affordable on CPU. Reranking hundreds of candidates at high QPS is not: there the linear cost explodes and the GPU wins. The rule of thumb: broad cheap recall in the retriever, precision rerank over few candidates. (The detail of hybrid retrieval and reranking is in the fundamentals piece linked below.)

The numbers, with honest methodology

Here comes the part where a lot of people lie by omission. I am going to give throughput ranges, but they are literature ranges and orders of magnitude, not my own measurements on this hardware. Take them as such: the right decision does not depend on nailing the number, it depends on understanding the allocation.

For bge-m3 dense, sequence ≈256 tokens, embedding throughput moves roughly like this:

Platform	Dense throughput (order of magnitude)	Reading
High-end GPU (5090 fp16, TEI)	~12k tok/s+ (indicative)	The ceiling; expensive and scarce.
Large server CPU (Xeon ~56 cores, int8 ONNX)	low band of thousands tok/s	~1/5–1/10 of the GPU, but horizontally scalable and cheap.
Edge CPU / NUC (4-8 cores, int8)	tens to low hundreds tok/s	Enough for overnight ingestion of a local corpus.

The temptation is to read the second row as “CPU is 5-10× slower, discarded”. That is the wrong reading for ingestion. For batch work with no SLA, what rules is not absolute tok/s but throughput per euro and throughput per watt, and there the arithmetic changes sign.

Take a numerical example of allocation. Suppose a corpus of 2 million chunks of around 256 tokens that has to be re-indexed once a day (the corpus changes, the embeddings have to be redone). That is:

$$2 \times 10^6 \text{ chunks} \times 256 \text{ tok/chunk} \approx 5.1 \times 10^8 \text{ tokens}$$

At a conservative CPU throughput of, say, 3,000 tok/s per int8 Xeon server:

$$t_{\text{ingest}} \approx \frac{5.1 \times 10^8 \text{ tok}}{3000 \text{ tok/s}} \approx 1.7 \times 10^5 \text{ s} \approx 47 \text{ hours on a single server}$$

47 hours on one box sounds bad until you remember two things. First, this is embarrassingly parallel: the corpus is sliced and distributed; with 8 CPU servers it drops to around 6 hours, with 16 to around 3 hours, comfortably inside the overnight window. Second, and more importantly: that same work on the GPU blocks the GPU. If the H100 does 12k tok/s, it takes around 12 hours, but they are 12 hours of the H100, the resource the whole organisation fights over in order to generate. Spending the scarce, expensive resource on re-indexing a corpus that changes once a day is bad allocation, even if it is “faster”: you are optimising the wrong tok/s.

The mental rule: for ingestion, optimise throughput/€ and throughput/W; absolute tok/s belongs to the generation plane, where the human stopwatch really is running.

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

The first case is a node with no GPU: a NUC, an office Xeon, a sovereign edge server in a branch or in an air-gapped environment. The whole data plane lives there; generation is delegated to a remote GPU endpoint or done in batch with an SLM when latency is not pressing.

The stack:

TEI-CPU serving bge-m3 int8 with dense + sparse (same OpenAI contract /v1/embeddings, plus /rerank for the reranker).
Qdrant with a dense HNSW index plus sparse vectors, native RRF fusion.
Reranker bge-reranker-v2-m3 over the top-k (via TEI’s /rerank).
Gateway that orchestrates and, for generation, calls an external endpoint.

# docker-compose: full RAG data plane on CPU (no GPU)
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-m3", "--pooling", "cls", "--dtype", "int8"]
 ports: ["8081:80"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX if the Xeon supports it

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-reranker-v2-m3", "--dtype", "int8"]
 ports: ["8082:80"]
 # exposes /rerank — invoked ONLY over top-20/50, never over hundreds

 qdrant:
 image: qdrant/qdrant:latest
 ports: ["6333:6333"]
 volumes: ["./qdrant_storage:/qdrant/storage"]
 # HNSW dense + sparse vectors + RRF, all CPU

Hybrid search with RRF fusion in Qdrant (dense + sparse in a single query):

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

# query embedding: dense and sparse from TEI-CPU (HTTP wiring omitted)
hits = client.query_points(
 collection_name="corpus",
 prefetch=[
 models.Prefetch(query=dense_vec, using="dense", limit=50),
 models.Prefetch(query=sparse_vec, using="sparse", limit=50),
 ],
 query=models.FusionQuery(fusion=models.Fusion.RRF), # native RRF
 limit=20,
).points
# -> then: POST /rerank (TEI) over these 20, keep the top-5
# -> then: the gateway sends query + top-5 to the GENERATION endpoint (GPU)

Generation, on this CPU-only node, leaves the node: the gateway builds the augmented prompt and sends it to a vLLM endpoint on the GPU cluster (or, if there is no interactive SLA, to an SLM on CPU in batch mode, accepting a TTFT of seconds). The entire data plane, everything above, runs without a single GPU.

Reference architecture (b): the recommended hybrid

This is the one I recommend for the general case with a GPU cluster available: data plane on CPU, generation plane on GPU, connected by OpenAI-compatible HTTP contracts so that each side is replaceable.

Generation server, minimal, on GPU:

# vLLM on the GPU cluster — generation ONLY
services:
 vllm-gen:
 image: vllm/vllm-openai:latest
 command: >
 --model meta-llama/Llama-3.1-8B-Instruct
 --dtype bfloat16 --max-model-len 8192
 --gpu-memory-utilization 0.85
 # exposes /v1/chat/completions — the gateway sends it query + the top-5 already retrieved
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]

The virtue of the design: because both sides speak the OpenAI contract over HTTP, the CPU data plane and the GPU generation plane scale separately and are replaceable. If tomorrow you want to move the rerank to GPU because QPS went up, you change one URL. If you want to add more CPU ingest nodes, you add them without touching generation. The whole stack is OSS and licence-clean: bge-m3 and bge-reranker-v2-m3 are MIT (bge-m3, reranker), Qdrant is Apache-2.0, TEI and vLLM are OSS.

Applied to the generic 4×H100 cluster

Let us bring this down to the cluster in the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct allocation:

Build and indexing → CPU fleet. No H100 should spend a cycle re-embedding the corpus. That goes to the Xeon AMX machines (large servers, throughput in the thousands of tok/s in int8) or, for small local corpora, to the NUCs overnight. Overnight re-indexing of a corpus that changes once a day is the textbook case of “unhurried CPU work”.
The H100s → generation. All four cards are reserved for what only they do well: producing tokens at interactive latency. This is what the sibling pieces in the series, sharing a GPU and several models on one GPU, help to squeeze: once ingestion stops competing for the GPU, all the expensive silicon is free to generate and is shared better between models and tenants.
The H100s, at most, → rerank peaks or large embedders. If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) for a domain where bge-m3 does not reach, or a massive rerank at a QPS the CPU cannot absorb, those peaks can indeed visit the GPU. But they are the occasional exception, not the base load.

The auditability angle: ENS / NIS2

There is a compliance argument that rarely gets mentioned and that the CPU/GPU split gives you almost for free.

A CPU-only node with no proprietary driver is easier to audit. There is no closed NVIDIA kernel stack, no CUDA and firmware versions to reconcile with the supply chain, no proprietary driver surface to document for an ENS or a NIS2 assessment. The whole data plane, chunking, embeddings, index, search, runs on OSS software on generic CPU with standard instructions. For a sovereign or classified environment, being able to say “the plane that touches the corpus depends on no proprietary binary” is a real argument, not marketing.

And there is a second auditability angle intrinsic to RAG done well: source traceability. A RAG system that retrieves identifiable chunks and cites them is auditable, since you can reconstruct which document each claim came from, as opposed to context-stuffing or the model’s opaque parametric knowledge, where there is no way to know where a fact came from. That traceability lives in the data plane (what was retrieved, from which source, with what score), precisely the plane we are putting on auditable CPU. The two arguments reinforce each other: the auditable silicon and the auditable chain of evidence are the same plane.

When NOT to take it to CPU

For honesty, and to avoid falling into the mirror image of the hype, the cases where CPU is not the answer:

Generation at interactive latency. The obvious case. A 7B on CPU gives TTFT in seconds: unacceptable for chat. If the user is waiting, generation goes to GPU. With no practical exceptions today.
Massive reranking at high QPS. A cross-encoder or ColBERT over hundreds of candidates, multiplied by many requests per second, saturates the CPU. The cost $\propto k \times \text{QPS}$ crosses the threshold where the GPU pays off. Keep CPU rerank bounded to top-20/50; if you need more breadth at higher QPS, move up to GPU.
Real-time re-indexing with a strict SLA. If the corpus changes continuously and freshness is measured in seconds (not hours), CPU throughput may not fit the window. There, ingest embedding may need a GPU, but note that this is rare: most corporate corpora change at the pace of hours or days, not seconds.
Large embedders (7B). bge-m3 (568M) is comfortable on CPU; a gte-Qwen2 or NV-Embed at 7B is LLM-class again and drags the same cost profile as generation. If your retrieval quality demands a 7B embedder, that embedder lives where 7Bs live: on the GPU.

The sentence that sums it all up: the CPU is the default home of the data plane; the GPU is the justified exception for what is latency-bound and massively online. Start by putting everything on CPU and move up to GPU only what proves it does not fit, not the other way round.

References

Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings. arXiv 2402.03216. https://arxiv.org/abs/2402.03216
BAAI — BGE-M3 model card (568M params, XLM-RoBERTa, 8192 tokens, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder on top of bge-m3, ~568M). https://huggingface.co/BAAI/bge-reranker-v2-m3
Intel + Hugging Face — CPU Optimized Embeddings with Optimum Intel and fastRAG (~10× indexing for BGE-large int8, 4th-gen Xeon). https://huggingface.co/blog/intel-fast-embedding
deepset / Haystack — CPU-Optimized Embedding Models with fastRAG and Haystack. https://haystack.deepset.ai/blog/cpu-optimized-models-with-fastrag
Hugging Face — Text Embeddings Inference (TEI), CPU ONNX/MKL backends, OpenAI-compatible endpoints. https://github.com/huggingface/text-embeddings-inference
Qdrant — fastembed (ONNX-CPU, dense/sparse/ColBERT) and Hybrid Search with RRF. https://github.com/qdrant/fastembed · https://qdrant.tech/documentation/beginner-tutorials/hybrid-search-fastembed/

Technical controls: the ENS × ISO 42001 × EU AI Act cross-mapping over the on-premise LLM architecture

Mon, 01 Jun 2026 06:30:00 +0200

Third post in the blog’s AI governance trilogy. The first, ISO/IEC 42001, broke down the certifiable management system. The second, EU AI Act, broke down the directly applicable legal regulation. This one covers the third framework that appears when the client is a Spanish public administration or an essential-services entity: the ENS (Esquema Nacional de Seguridad, Spain’s national security framework, Royal Decree 311/2022). The editorial challenge: triple compliance is not the arithmetic sum of three jobs; it is one single set of technical evidence labelled for three lenses. This post takes apart how that evidence is built, measure by ENS measure.

TL;DR

Royal Decree 311/2022 updated the Spanish Esquema Nacional de Seguridad (ENS), aligning it with NIS2 and with the modern cyber-threat landscape. It applies mandatorily to the entire Spanish public sector (central, regional and local administration, universities, autonomous bodies) and to suppliers that provide IT services to them. The framework defines 74 security measures organised into three blocks (organisational framework org, operational framework op, protection measures mp), three application categories (Basic / Medium / High) according to the assessment of five dimensions (Confidentiality, Integrity, Traceability, Authenticity, Availability), and an Annex II with the measure × category matrix that dictates what is required at each level. This post maps the technical measures relevant to LLM systems (those of the op block: planning, access control, operation, external services, continuity, monitoring; and those of the mp block: communications, applications, information, services) against Annex A of ISO/IEC 42001 and the operational articles of the EU AI Act (Arts. 9 to 15, 17, 72 and 73), and shows that the overlap is massive: a single technical artefact from the blog’s OSS stack, the OTel spans of tracing, the datasets versioned with DVC, the LLM Guard scanners, the guardrail decisions, the incidents of the retrain loop, simultaneously satisfies ENS measures, 42001 controls and AI Act articles, provided it is labelled with the correct codes from the moment it is captured. The editorial thesis: the difference between a system that passes all three audits and one that suffers three separate certifications is not budget, it is disciplined labelling and a common vocabulary. The post builds the master table of triple compliance, walks through the case of the multi-tenant chatbot for a High-category public administration as a live checklist, and closes with the five traps of triple compliance.

The analogy: the triple inspection of a critical building

A critical building, say the control centre of a regional electricity grid, a reference hospital, or a 112 emergency response centre, goes through three different inspections of the same physical elements:

The regional inspection reviews the planning permit, local fire regulations and accessibility under regional legislation.
The European inspection reviews CE marking of equipment and conformity with EU directives on energy efficiency.
The ISO quality inspection reviews processes and maintenance under a management standard.

All three inspectors look at the same fire detector hanging from the ceiling. The regional one wants to see the installation certificate with the installer company’s number; the European one wants to see the CE mark stamped on the casing; the ISO one wants to see the monthly maintenance record with signatures. There is only one detector. The evidence each inspector needs is different. If the building management team kept one folder per inspector, the installation certificate, the photo of the CE mark and the maintenance sheet would live in three places, and on audit day inconsistencies between the three folders would surface. The professional way to operate: one single file per asset, with tags pointing at the three sets of legislation.

The on-premise LLM system is that critical building. The OTel spans from tracing are the fire detector: a single technical piece that satisfies ENS measure op.exp.8 (activity logging), 42001 control A.8.2 (information to interested parties), and EU AI Act article 12 (record-keeping) provided the spans carry the correct metadata: traceparent propagated, gen_ai.* semantic conventions, WORM retention, PII redacted by LLM Guard Vault. Without consistent labelling from capture onwards, audit day produces three folders that do not add up.

The analogy matters because it bounds an architectural decision: labelling evidence is not a compliance topic, it is a technical design topic. It is decided when the OTel Collector is set up, when the Vault schema is designed, when the format of retrain incidents is agreed. Postpone it, and the debt compounds.

ENS in 15 seconds

Regulatory framework: Royal Decree 311/2022, of 3 May, in force since 5 May 2022. It replaces RD 3/2010. It implements the update of the Esquema with NIS2, the NCS (Estrategia Nacional de Ciberseguridad, Spain’s national cybersecurity strategy) and the post-2020 reality.
Mandatory scope: the Spanish public sector (central, regional and local administration, universities, autonomous bodies), public-sector entities with their own legal personality, and private suppliers that provide IT services to them (a very important clause in consultancy: if your client is the Junta de Andalucía, the Comunidad de Madrid or Bilbao City Council, you fall under ENS by contractual extension).
Operational applicability in 2026: any AI project financed with NextGenerationEU European funds, plus any project with public data, plus any integration with electronic administrative systems.
Categorisation: three categories, Basic / Medium / High, according to the assessment of the five dimensions (C, I, T, A, D). The category dictates which measures are required and at what depth.
Certification: for Medium and High categories, a formal audit by an accredited body. For Basic category, self-assessment. Certification cycle is biennial.
Authority: the Centro Criptológico Nacional (CCN-CERT, Spain’s national cryptologic centre), part of the CNI intelligence agency. It maintains the ens.ccn.cni.es portal with the STIC guides (800 series).
Total number of measures: 74, organised into three blocks, plus Annex II with the measure × category matrix that defines what applies at each level.

The five security dimensions and their AI mapping

To classify the system as Basic / Medium / High, the ENS requires each dimension to be rated on a scale (not applicable / low / medium / high). The final category is the highest of the five. For LLM systems, the typical rating:

Dimension	Meaning	Typical LLM rating	Reason
C Confidentiality	Information protected from disclosure	Medium-High	PII in prompts, secrets in context, intellectual property in the RAG corpus
I Integrity	Information protected from modification	Medium-High	An altered RAG corpus produces false answers; a manipulated model produces directed bias
T Traceability	Actions attributable to users	High	Audit: who asked what, when, what were they answered, which dataset trained the model?
A Authenticity	Identity of users and origin of information	Medium	Robust authentication plus identification of chunks by source
D Availability	Service available when needed	Medium	Typical SLA, known recovery time

Typical result: Medium category for internal LLM systems with no sensitive data, High if they handle PII or regulated sectoral data (health, tax, judicial). The category triggers different controls.

The three blocks of measures

Organisational framework (org), 4 measures that always apply, transversal:

org.1 Security policy
org.2 Security regulations
org.3 Security procedures
org.4 Authorisation process

Operational framework (op), 31 measures in 6 subgroups, these are how it is operated:

op.pl Planning (5)
op.acc Access control (6)
op.exp Operation (10)
op.ext External services (4)
op.cont Service continuity (4)
op.mon System monitoring (2)

Protection measures (mp), 39 measures in 7 subgroups, these are what is protected:

mp.if Facility protection (7)
mp.per Personnel management (4)
mp.eq Equipment protection (4)
mp.com Communications protection (4)
mp.si Information media protection (5)
mp.sw Application protection (2)
mp.info Information protection (6)
mp.s Service protection (4)

Of those 74, roughly 25 matter operationally for LLM systems. The rest are either transversal (policies, personnel management, physical facilities) or specific to other layers (telephony, physical media in the literal sense). The rest of this post drops down to the 25 relevant ones.

Technical mapping: key ENS measures × 42001 controls × EU AI Act articles

What follows, by ENS subgroup, are the measures relevant to LLM with their cross-mapping. The Technical evidence from the blog column points to the operational artefact that materialises the measure.

Operational framework — planning (`op.pl`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.pl.1` Risk analysis	Formal periodic analysis, MAGERIT or OCTAVE methodology	A.5.4 (alignment with AI risk treatment)	Art. 9 Risk management	Risk document linked to the LLMOps pipeline + evals metrics
`op.pl.2` Security architecture	Architecture documentation, layer segregation	A.4.2 documented info	Art. 15 (technical robustness)	Seven stack layers + seven deployment phases
`op.pl.3` Acquisition	Acquisition criteria that include security	A.10.3 suppliers	Art. 53 (GPAI obligations)	OSS vs hyperscalers lock-in analysis + copyright analysis
`op.pl.4` Sizing	Capacity to support the expected load	A.4.5 system resources	Art. 15 (consistent performance)	Five maturity levels + GPU capacity study
`op.pl.5` Certified components	Preference for certified components	A.10.5 third parties	Art. 53 + Art. 15	OSS inventory with licence analysis + supply chain audit (cosign + SLSA)

Operational framework — access control (`op.acc`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.acc.1` Identification	Unique identification of users and processes	A.3.2 roles	Art. 14 (human oversight)	Keycloak / OIDC + JWT with unique `sub` + `user_id_hashed` in OTel spans
`op.acc.2` Access requirements	Need to know, authorisation	A.9.4 intended use	Art. 14 + Art. 26 (deployer)	Per-tenant allowlist in the AI Gateway + RBAC over adapters
`op.acc.3` Segregation of duties	No incompatible accumulation	A.3.2 + A.3.3	Art. 17 (QMS)	Separate roles: AI lead / data steward / SRE / DPO
`op.acc.5` Authentication	Proportionate mechanism, MFA at High	A.4.4 tooling	Art. 14	Keycloak + WebAuthn + mandatory MFA for High category
`op.acc.6` Local access	Protection against physical access	mp.if + A.4.5	Art. 15 (cybersec)	Cluster in a datacenter with physical control (strictly outside the blog’s scope)
`op.acc.7` Remote access	VPN, encryption, endpoint control	A.4.5 + mp.com	Art. 15	WireGuard / Defguard + mTLS for administrative access to the cluster

Operational framework — operation (`op.exp`)

This is the densest subgroup and the one that overlaps most with the LLM architecture.

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.exp.1` Asset inventory	CMDB with all system components	A.4 resources	Art. 49 (EU DB registration)	Inventory of models, adapters and datasets in a CMDB + Helm tags + OpenLineage lineage
`op.exp.2` Security configuration	Documented hardened configuration	mp.eq + A.4	Art. 15	vLLM, KServe and Cilium configurations documented in GitOps + CIS Benchmarks for K8s
`op.exp.3` Configuration management	Traceable and authorised changes	A.6.2 + A.4.2	Art. 9 + Art. 15	GitOps with Argo CD / Flux + mandatory PR review + immutable tags
`op.exp.5` Change management	Changes planned, authorised, recorded	A.6.2.6 + clause 8	Art. 9 (lifecycle) + Art. 72 (post-market)	CI/CD pipeline from the LLMOps post + change advisory board
`op.exp.6` Protection against malicious code	Antivirus, EDR, malware control	mp.eq	Art. 15 (cybersec)	Image scanning (Trivy / Grype) + container runtime security (Falco / Tetragon)
`op.exp.7` Incident management	Detection → response → recovery procedure	A.3.3 + clause 10	Art. 73 (serious incident reporting)	Incident-driven retrain + CCN-CERT channel + notification template
`op.exp.8` Activity logging	Auditable logs, minimum 2-year retention at High category	A.8.2	Art. 12 + Art. 19	OTel GenAI tracing + Tempo / Jaeger + Loki
`op.exp.9` Incident management records	Incident log with root cause and corrective action	clause 10	Art. 73	Ticketing system + structured retrain incident events
`op.exp.10` Record protection	Immutable logs, cryptographic integrity, guaranteed retention	A.8.2 + A.4.2	Art. 12	WORM storage (Ceph + immutable bucket) + log signing (sigstore) + 24-36 month retention
`op.exp.11` Cryptographic keys	Key lifecycle management, HSM at High category	mp.info.4	Art. 15	HashiCorp Vault / SOPS + HSM (Yubico, AWS KMS on-prem) for High category

Operational framework — external services (`op.ext`)

Critical when the system integrates GPAI (Llama, Mistral) hosted elsewhere, or when the deployer is a third party.

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.ext.1` Contracting external services	Contract with security clauses, SLA, right to audit	A.10.3 suppliers	Art. 25 + Art. 53	Contract with ENS clauses + annual audit review + Cloud Act analysis
`op.ext.2` Alternative means	Plan B if the supplier goes down	A.4.5 + clause 6	Art. 15 (resilience)	Multi-cluster with failover + qualified alternative GPAI (Llama→Mistral→Qwen)
`op.ext.3` Supply chain protection	Assessment of suppliers and sub-suppliers	A.10 + clause 6	Art. 53 + NIS2 supply chain	SBOM + cosign + SLSA + continuous vulnerability scanning
`op.ext.4` System interconnection	Interconnection agreements, secure gateways	mp.com + A.4.5	Art. 15	API Gateway with mTLS + JWT signing + rate limiting + WAF

Operational framework — continuity (`op.cont`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.cont.1` Impact analysis	BIA per system	A.5.5 (impacts on individuals)	Art. 9 (risk management)	Documented BIA with RPO / RTO per system
`op.cont.2` Continuity plan	Documented DRP, RTO/RPO	clause 6 + A.4	Art. 15 (resilience)	DRP + Velero K8s backups + DVC datasets in a secondary bucket
`op.cont.3` Periodic testing	Drills at a defined frequency (annual at High category)	clause 9 (evaluation)	Art. 15	Annual game-day with a simulated disaster + recovery stopwatches
`op.cont.4` Alternative means	Capacity to continue with degraded means	clause 6 + A.4.5	Art. 15	Secondary cluster in a different datacenter + reserved GPU pool + replicated datasets

Operational framework — monitoring (`op.mon`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.mon.1` Intrusion detection	IDS/IPS over network and applications	A.9.2 use	Art. 15 (cybersec)	Guardrails as a semantic WAF + LLM Guard PromptInjection + Tetragon eBPF runtime
`op.mon.2` Metrics system	Measurable operational metrics, auditable dashboard	clause 9 (performance evaluation)	Art. 72 (post-market monitoring)	Prometheus + VictoriaMetrics + Grafana + Langfuse dashboards
`op.mon.3` Surveillance (High category)	24×7 monitoring with alerting	clause 9 + clause 10	Art. 72 + Art. 73	SOC with SIEM alerts (Wazuh, OpenSearch, Vector + custom) + on-call rotation

Protection measures — communications (`mp.com`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.com.1` Secure perimeter	Firewall, segmentation, DMZ	A.4.5	Art. 15	Cilium NetworkPolicy + Calico + ingress controllers with WAF (mod_security)
`mp.com.2` Confidentiality protection	Encryption in transit (TLS 1.2+ mandatory, 1.3 recommended)	A.4.5	Art. 15	TLS 1.3 mandatory + cert-manager + Let’s Encrypt or an internal CA
`mp.com.3` Integrity and authenticity protection	Encryption + origin authentication	A.4.5 + A.4.4	Art. 15	Intra-cluster mTLS + JWT signing at the gateway + checksums on artifacts
`mp.com.4` Flow separation	Segmentation of mgmt vs production vs external traffic	A.4.5	Art. 15	Cilium policies + network namespaces + east-west / north-south segregation

Protection measures — applications (`mp.sw`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.sw.1` Application development	Secure SDLC, code review, SAST/SCA	A.6.2.3 responsible design	Art. 9 + Art. 15	Forgejo CI with SAST (Semgrep, CodeQL) + SCA (Trivy, Grype) + mandatory PR review
`mp.sw.2` Acceptance and go-live	Acceptance tests, eval gates before production	A.6.2.5 V&V	Art. 9 + Art. 15	Eval gates from the evals post + canary deploy + pre-go-live metrics

Protection measures — information (`mp.info`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.info.1` Personal data	GDPR compliance + technical and organisational measures	A.5.5 + A.7.6	Art. 10 + Art. 26	LLM Guard Vault + Presidio + minimisation in RAG corpus curation
`mp.info.2` Information classification	Labelling by level (public / internal / confidential / restricted)	A.7.2 data	Art. 10	Schema contracts from data versioning with a `classification` field
`mp.info.3` Encryption	At-rest minimum at Medium category, High category with HSM	A.4.5 + mp.eq	Art. 15	LUKS / dm-crypt on disks + encryption in the Ceph bucket + keys in Vault
`mp.info.4` Electronic signature	Documents signed with a valid certificate (Medium category and above)	A.8.2	Art. 12	Log signing with sigstore + signing of published models with cosign
`mp.info.5` Timestamps	Qualified timestamp for temporal integrity (High category)	A.8.2	Art. 12	Timestamping with a qualified TSA + RFC 3161 on critical events
`mp.info.6` Document cleaning	Removal of unauthorised metadata, anonymisation	A.7.6 + A.5.5	Art. 10	LLM Guard Anonymize (input) + Sensitive (output) + Vault with TTL

Protection measures — services (`mp.s`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.s.1` Email	Anti-spam, anti-malware, optional encryption	A.4.5	—	Outside the direct LLM scope
`mp.s.2` Protection of web services and applications	WAF, hardening, vulnerability management	A.9.2 + mp.com	Art. 15 (cybersec)	AI Gateway (LiteLLM / Envoy AI / Kong AI) with policies + ModSecurity + Cloudflare-like
`mp.s.3` Protection against denial of service	Rate limiting, anti-DDoS, capacity planning	A.4.5 + A.9.2	Art. 15	Rate limiting at the gateway + token quotas + circuit breakers
`mp.s.4` Protection against external threats (High category)	Advanced monitoring, threat intel	A.9.2 + clause 9	Art. 15 + Art. 72	Guardrails, 4 lines + advanced LLM Guard scanners + threat intel feed (CCN-CERT MISP)

Master table of triple compliance — the 25 relevant controls consolidated

The key reading of the table: one row per technical capability. The organisation does not build three solutions for “activity log” (ENS) + “information to interested parties” (42001) + “record-keeping” (AI Act). It builds one single piece of OTel tracing with the correct metadata and presents it labelled for whichever inspector is asking.

The labelling typically materialises in three mechanisms:

Tags in the CI/CD pipelines: every artefact produced carries tags ens:op.exp.8, iso42001:A.8.2, aia:art.12 in its Helm / Argo CD metadata.
Semantic OTel attributes: gen_ai.compliance.ens = "op.exp.8", gen_ai.compliance.iso42001 = "A.8.2" as custom attributes on the relevant spans (not standard yet, but useful internally).
A mapping table in the wiki: a living document of ENS measure → 42001 control → AI Act article → technical runbook + owner + last verification. This is the artefact the auditor consults.

Applied case: multi-tenant chatbot for public administration

A variant of the multi-tenant chatbot from the forensic post, where the client is now a Spanish regional government offering citizen assistance on administrative procedures. The three lenses:

ENS: applies mandatorily, being public sector plus a service to citizens. Category High (citizens’ PII plus a critical service).
ISO 42001: the regional government requires certification from the supplier as a contractual condition.
EU AI Act: if the chatbot informs about procedures but decides nothing on the citizen’s behalf, it is limited risk (Art. 50 transparency). If it automates decisions (admission to a programme, refusal of a benefit), it is high risk.

We assume high risk for the most demanding walkthrough.

The chatbot’s 25 key technical capabilities consolidated

Technical capability	ENS	42001	AI Act	Status
Robust citizen authentication	`op.acc.5` (MFA High category)	A.4.4	Art. 14	Cl@ve + digital certificate + WebAuthn
Unique tenant + user identification	`op.acc.1`	A.3.2	Art. 14 + 26	JWT with `tenant_id` + `user_id_hashed`
Per-tenant allowlist	`op.acc.2`	A.9.4	Art. 14	LiteLLM Proxy with policies + Envoy filter
Encryption in transit, mTLS	`mp.com.2-3`	A.4.5	Art. 15	TLS 1.3 + cert-manager + intra-mesh mTLS
Encryption at rest + HSM keys	`mp.info.3` + `op.exp.11`	A.4.5	Art. 15	LUKS + Vault + Yubikey or nCipher HSM
PII anonymisation input/output	`mp.info.1 + .6`	A.7.6	Art. 10	LLM Guard Vault + Presidio + Llama Guard 4
Traceable logging + WORM	`op.exp.8 + .10`	A.8.2	Arts. 12 + 19	OTel + Tempo + Loki + Ceph WORM 36 months
Log signing + timestamps	`mp.info.4 + .5`	A.8.2	Art. 12	sigstore + qualified TSA (FNMT)
Runtime threat detection	`op.mon.1 + mp.s.4`	A.9.2	Art. 15	Guardrails, 4 lines + Tetragon + Falco
Operational metrics dashboard	`op.mon.2`	clause 9	Art. 72	Prometheus + Grafana + Langfuse
24×7 SOC surveillance	`op.mon.3` (High category)	clause 9	Art. 72	SOC with SIEM + on-call rotation
Incident management with notification	`op.exp.7 + .9`	clause 10	Art. 73	Incident-driven retrain + CCN-CERT channel
CI/CD pipeline with eval gates	`mp.sw.1 + .2`	A.6.2.3-5	Arts. 9 + 15	Forgejo CI + Semgrep + Trivy + DeepEval gates
Backups + DRP	`op.cont.1-4`	clause 6 + A.4	Art. 15	Velero + DVC datasets in a secondary bucket + annual game-day
Systematic risk analysis	`op.pl.1`	A.5 + cl.6	Art. 9	MAGERIT + FRIA + ISO/IEC 23894 impact assessment
Documented secure architecture	`op.pl.2`	A.4.2	Art. 15	Seven layers + seven phases
Supplier management	`op.ext.1 + .3`	A.10.3	Art. 53	Contracts with ENS clauses + SBOM + Cloud Act analysis per GPAI
Certified components	`op.pl.5`	A.10.5	Art. 53	Inventory with licence + supply chain audit
Configuration hardening	`op.exp.2 + .3`	A.4 + A.6	Art. 15	CIS Benchmarks for K8s + GitOps + immutable tags
Protection against malicious code	`op.exp.6`	mp.eq	Art. 15	Image scanning + runtime security
Asset inventory	`op.exp.1`	A.4	Art. 49	CMDB + Helm tags + OpenLineage
Network segmentation	`mp.com.4`	A.4.5	Art. 15	Cilium NetworkPolicy + namespaces
Rate limiting + anti-DoS	`mp.s.3`	A.4 + A.9	Art. 15	LiteLLM rate limit + token quotas
Information classification	`mp.info.2`	A.7.2	Art. 10	Schema contracts with a `classification` field
Transparency towards the user	—	A.9.4	Art. 50	UI banner + disclaimer in responses

Result: the 25 technical capabilities are common to all three frameworks. Only two require evidence for a single framework in isolation: mp.info.4-5 (qualified signature and timestamp, ENS-specific, not required so literally by 42001 or the AI Act) and Art. 50 (transparency banner, AI Act-specific). The rest are the same technical piece with three labels. A well-organised team certifies all three in series with only marginal extra work between the second and the third.

The five traps of triple compliance

Trap 1 — Measuring the same thing three times. Inexperienced teams create three separate dashboards (one for ENS, another for 42001, another for the AI Act) with the same metrics duplicated. The result: three sources of truth that diverge, three audit teams with different figures, three corrections to resolve one and the same deviation. The rule: one metric, three labels.

Trap 2 — Losing the control that covers only one framework. mp.info.4 (electronic signature) is ENS-specific and gets forgotten while the team is focused on 42001 + AI Act. On ENS audit day, the gap surfaces. Solution: the master table keeps every control visible, orphans included.

Trap 3 — Bias towards the most recent framework. The team spends 80% of the effort on the AI Act because it is the newest, and forgets the rigour of the ENS, which has been in force for 14 years. The ENS measures are technically more prescriptive than the AI Act (which is legally stricter but leaves implementation freedom). Moving up a level to ENS High category introduces specific requirements (HSM, qualified timestamps, 24×7 surveillance) that the AI Act does not spell out. The ENS deserves respect for its technical detail, not for its legal stature.

Trap 4 — Mixing ENS Medium and High categories. The Annex II matrix of RD 311/2022 dictates which measures are required and at what depth per category. Moving from Medium to High changes 15-20 controls, which is not marginal. The category is decided at the start of the project and documented; changing it halfway forces a complete re-audit.

Trap 5 — Audit fatigue inside the technical team. Three audits a year (biennial ENS + annual 42001 surveillance + ad-hoc AI Act by the authority) exhausts the team if they are not planned and if evidence is not reused. The professional approach: one single quarterly internal audit cycle with rotating scope, producing evidence consumable by all three external auditors. The difference between 30 days a year and 90 days a year lost to audits is the discipline of single evidence plus disciplined labelling.

What we have not covered (upcoming posts)

The CCN-CERT STIC series applicable to AI systems: STIC 800-159 (web service operation), STIC 800-105 (cryptography), STIC 800-150 (cloud environments). Each adds technical detail on how to materialise the ENS measures.
The GDPR as a fourth lens: privacy and personal data protection. It overlaps with mp.info.1 and with A.7.6 of 42001 plus Art. 10 of the AI Act. Material for an analogous pass over LOPDGDD + GDPR + AEPD against the three frameworks seen here.
Concrete templates of technical evidence with minimum fields: a log entry with every attribute required by the three lenses, an incident report with every field demanded by the three frameworks, a unified ENS / AI Act / 42001 declaration of conformity.
The NextGenerationEU funds case: specific compliance requirements for AI projects financed with European funds, where the AI Act + ENS are contractually mandatory.
42001 + ENS High category combined with DORA (Digital Operational Resilience Act, Reg. 2022/2554) for Spanish financial entities deploying AI.

References

RD 311/2022 — por el que se regula el Esquema Nacional de Seguridad. https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
CCN-CERT — Portal ENS: https://ens.ccn.cni.es/. Guías STIC Series 800.
CCN-STIC 803 — Valoración de los sistemas y de la información. Metodología para asignar categoría ENS.
CCN-STIC 804 — Esquema Nacional de Seguridad. Guía de implantación.
CCN-STIC 824 — Informe del estado de seguridad. Plantilla para auditoría ENS.
MAGERIT v3 — Metodología de Análisis y Gestión de Riesgos del Ministerio de Asuntos Económicos. Insumo de op.pl.1.
ISO/IEC 42001:2023 — Sistema de gestión IA. Norma certificable que solapa con ENS.
Regulation (EU) 2024/1689 (EU AI Act) — texto consolidado.
NIS2 (Dir. 2022/2555) — directiva de ciberseguridad que el ENS implementa parcialmente en su versión 2022.

ISO/IEC 42001: the operations manual for the AI system — how the AIMS fits over the on-premise LLM platform described in this blog

Mon, 01 Jun 2026 06:00:00 +0200

This post closes an asymmetry the blog had been accumulating: we have described in detail the technical platform (seven stack layers, seven deployment phases, five maturity levels), the operational pipeline (six LLMOps stages), the data pieces (corpus curation, versioning), the eval / safety pieces (evals, guardrails, LLM Guard) and the observe pieces (OTel GenAI tracing). What had not appeared is the governance layer a regulated client asks for on top of all that. ISO/IEC 42001 is that layer.

TL;DR

ISO/IEC 42001:2023 is the first certifiable international standard that defines how an AI system is managed. It is not a technical standard (it does not say “use this inference engine” or “this safety threshold”): it is a management standard, a cousin of ISO 27001 and ISO 9001. From both it inherits the Annex SL structure, seven mandatory clauses running through context, leadership, planning, support, operation, performance evaluation and improvement, and it adds an Annex A with 38 AI-specific controls in 9 sections: policies, internal organisation, resources, impact assessment, lifecycle, data, information to interested parties, use, third parties. The thesis of this post is that the technical architecture described in this blog directly covers between 60% and 80% of the A controls with no extra work. The LLMOps pipeline materialises A.6, versioning and curation materialise A.7, guardrails and evals materialise A.9, OTel tracing materialises A.8. The rest is governance discipline that does not appear in the code (a written AI policy, impact assessments per system, a stakeholder register, role decisions between provider/producer/customer, mandatory documentation), and that is precisely what separates a real certification from performative compliance. The post maps the correspondence control by control, cross-references it with the EU AI Act (seven articles directly aligned with 42001: 9, 10, 11, 12, 13, 14, 17), with NIS2 (asset register, incident notification, supply chain) and with ENS (Esquema Nacional de Seguridad, Spain’s national security framework, RD 311/2022, Basic/Medium/High categories), lists the seven mandatory minimum documents an auditor asks for, presents the blog’s multi-tenant chatbot as a live 42001 checklist, and closes with the five usual traps (confusing 42001 with EU AI Act compliance, over-documenting without measuring, ignoring A.5 until the audit, assuming 27001 covers the AI part, thinking certification is a one-off project rather than a living system).

The analogy: the aircraft operations manual

A modern aircraft, an A350, a Boeing 787, a drone certified for urban delivery, does not fly because it has good engines. It flies because the organisation that operates it has an Operations Manual approved by the aviation authority (EASA in Europe, the FAA in the US, AESA in Spain as a delegated body). The manual does not contain the engine blueprints, which the manufacturer certifies; it contains the procedures: who is captain on each flight, which checklist is run before each take-off, which periodic inspections happen at 100, 500 and 2,000 flight hours, which external suppliers are authorised to touch which components, what is documented after each incident, what to do when a new alert appears on the panel. The authority does not sit in on every flight: it reads the manual, randomly audits the traceability of past flights against the manual, and if everything adds up, it keeps the certification in place.

An AI system in production, the multi-tenant chatbot from the forensic post, a copilot for lawyers, a credit scoring system, is exactly the same. It flies because the model is good, the LLMOps pipeline is well built, the guardrails catch the bad cases. But it certifies because the organisation operating it has an AIMS (AI Management System) described in an auditable manual. ISO/IEC 42001 is that manual: its mandatory table of contents (Annex SL, seven clauses) and its catalogue of AI-specific controls (Annex A, 38 controls). The auditor does not sit next to the MLOps engineer: they read the AI policy, review the impact assessments of the most recently deployed systems, check that incident retrain is documented, verify third-party contracts, audit a sample of Langfuse traces cross-checked against dataset_hash and prompt_id. And if everything adds up, they certify.

The analogy matters because it frames the right question: 42001 does not certify the model or the code. It certifies the way of operating the complete system. A team can have the best OSS stack in the world and fail the audit because it has no written AI policy and no documented decision about which role (provider vs producer vs customer) it occupies towards its clients. And the reverse: a team with a modest model but with operations-manual discipline can certify without acrobatics.

ISO/IEC 42001 in 15 seconds

Publication: December 2023, ISO/IEC JTC 1/SC 42 (the ISO/IEC subcommittee for AI).
Status in 2026: an active standard, certifiable by accredited bodies (BSI, AENOR, TÜV, Bureau Veritas, A-LIGN, Schellman). Not yet formally recognised as a harmonised standard under the EU AI Act, but it provides the management base to lean on.
Compatibility: it shares the Annex SL structure with ISO 9001 (quality), 27001 (information security), 27701 (privacy), 22301 (continuity), 20000-1 (IT services). Organisations with integrated management systems (IMS) add it with 20-40% of the effort it would take to implement from scratch.
Applicability: any organisation that develops, provides, deploys or uses AI systems. It is not limited to developers: a company that consumes a hosted LLM and integrates it into its own product is within scope.
Certification: a 3-year cycle with an initial audit (Stage 1: documentation review + Stage 2: on-site audit) and annual surveillance audits. Typical cost: 15,000-60,000 € for the initial one depending on size; 6,000-20,000 € per annual surveillance.

What 42001 does not do:

It does not say which models to use or which thresholds to apply.
It does not certify the individual model (that is done by specific evaluations such as a NIST AI RMF profile or EU AI Act technical documentation).
It does not replace the EU AI Act or the GDPR: it is complementary. Implementing it well makes legal compliance easier but does not guarantee it.
It is not a technical standard for explainability or robustness (those are ISO/IEC 25059, 24029, 23894 and others in the SC 42 family).

Distinction from neighbouring frameworks

Framework	Nature	Scope	Certifiable	Overlap with 42001
ISO/IEC 42001:2023	Management standard	AIMS for any AI system	Yes	—
EU AI Act (Reg. 2024/1689)	Binding legal regulation	AI systems in the EU, risk-categorised	No (it is law)	Arts 9, 10, 11, 12, 13, 14, 17
NIS2 (Dir. 2022/2555)	Cybersecurity directive	Essential/important entities	Via the national scheme	Asset register, incident, supply chain
ENS (RD 311/2022)	Spanish security regulation	Public sector and its suppliers	Yes (B/M/A categories)	Traceability, incident management
ISO/IEC 27001	Management standard	Information security	Yes	Annex SL structure + Annex A overlap
ISO/IEC 27701	Management standard	Privacy (extends 27001)	Yes	PII in training data
NIST AI RMF 1.0	Voluntary framework	AI risk management	No	Conceptually aligned, not identical
ISO/IEC 23894	Technical standard	AI risk management	No	Input to A.5 (impact assessment)
ISO/IEC 5259	Family	Data quality for AI	No	Input to A.7 (data)

Three distinctions that matter operationally and are a recurring source of confusion with clients:

ISO 42001 ≠ EU AI Act compliance. Holding the 42001 certification makes it easier to demonstrate articles 9-17 of the European Regulation, but the Regulation requires things 42001 does not cover directly (CE marking of high-risk systems, registration in the European database, declaration of conformity, specific post-market monitoring). Implementing 42001 first and then filling the AI Act gaps is the standard route.
ISO 27001 is not enough. 27001 covers confidentiality, integrity and availability of information. The AI side is missing: bias, opacity, model drift, training corpus quality, human evaluation, impact on affected persons. 42001 is a complement, not a substitute. Organisations that already have 27001 in place have an advantage because they share half the documentation.
NIS2 ≠ AI safety. NIS2 requires registering critical assets, notifying incidents within 24 h, and managing the digital supply chain. AI systems can be within NIS2 scope if they form part of the critical asset (an LLM serving customer support in a financial entity is), but NIS2 does not audit model quality. 42001 does.

The seven clauses (Annex SL): the mandatory table of contents

The seven clauses, from clause 4 to clause 10, are common to all modern management standards (Annex SL, also called the “High Level Structure”). That means an organisation with ISO 9001 or 27001 already in place recognises the structure. Clauses 1-3 are introductory (scope, normative references, terms).

Clause 4 — Context of the organisation

Identify the external context (applicable regulation, customer expectations, social risks) and the internal context (strategy, culture, capabilities). Identify the interested parties and their expectations: customers, regulators, affected persons, employees, suppliers. Define the scope of the AIMS: which AI systems are in and which are out.

The usual gap: organisations that say “all our AI systems are in scope” without having enumerated them. The auditor asks for the list. Without a list, there is no scope.

Clause 5 — Leadership

Top management must approve and publish an AI policy, assign roles and responsibilities (typically AI lead, AI risk owner, data officer), and demonstrate commitment through resources, communication and oversight. The policy is an auditable document and must be made available to staff and interested parties.

The usual gap: a generic AI policy copied off the internet, with no measurables and no concrete objectives. The auditor asks how compliance with it is measured. Without metrics, the policy is theatre.

Clause 6 — Planning

Identify risks and opportunities of the AIMS (not of the individual model). Define measurable AI objectives, with deadlines and owners. Plan changes to the AIMS.

The usual gap: confusing AIMS risks (what happens if we do not document correctly?) with model risks (what happens if the model is biased?). The first belongs here; the second goes to A.5.

Clause 7 — Support

Resources: human, technical, financial, infrastructure. Competence of personnel (documented training). Awareness of the policy among personnel. Internal and external communication. Documented information (the backbone of the management system: policy, procedures, records, evidence).

The usual gap: documentation scattered across Confluence/Notion/Drive with no version control and no recorded approvals. The auditor asks about the last change: who approved it? when? on what grounds?

Clause 8 — Operation

The most operational clause. It requires:

Operational planning and control: how the AI system’s lifecycle is managed day to day. → Covered in the blog by the six-stage LLMOps pipeline.
Impact assessment (linked to A.5).
AI system lifecycle management (linked to A.6).
Data for AI systems (linked to A.7).

This is the clause that materialises in controls A.5, A.6 and A.7. On its own it adds no new requirements: it links to Annex A.

Clause 9 — Performance evaluation

Monitoring, measurement, analysis, evaluation. Internal audits (planned, with criteria, scope, frequency, recorded results). Management review (typically quarterly or half-yearly, with a mandatory agenda: inputs, evidence, decisions, actions).

The usual gap: there is OTel tracing + Langfuse + Grafana and plenty of data, but there is no formal management review agenda with documented minutes. The auditor asks for the minutes. Without minutes, there is no review.

Clause 10 — Improvement

Nonconformity and corrective action: when something fails, it is recorded, root cause is analysed, a correction is agreed, effectiveness is verified. Continual improvement: the system evolves deliberately.

The usual gap: Jira tickets with technical post-mortems but no formal “ISO nonconformity” record closed out with an effectiveness check. They are two distinct artefacts, even if they can be integrated.

The 38 Annex A controls: the AI-specific catalogue

Unlike Annex SL (which is common), Annex A is 42001’s AI-specific signature. The 38 controls are organised into 9 sections (A.2 to A.10; A.1 is the introduction) covering the AI-specific risks: opacity, bias, drift, corpus quality, impact on affected persons, third-party dependency. Each control has an objective (what it aims to achieve) and implementation guidance in Annex B.

Section	Focus	# controls
A.2	AI-related policies	2
A.3	Internal organisation	3
A.4	Resources for AI systems	6
A.5	Impact assessment	5
A.6	AI system lifecycle	4
A.7	Data for AI systems	5
A.8	Information for interested parties	4
A.9	Use of AI systems	3
A.10	Third parties and customer relationships	4
Total	—	38

What follows is the section-by-section mapping of the controls onto material we have already covered in the blog. The editorial intent is to show which gaps remain once the technical architecture is in place, so that the road to certification does not start from zero.

Cross-mapping: 38 controls ↔ blog posts

A.2 — AI policies (2 controls): PARTIAL

A.2.2 AI policy: the organisation must have a documented AI policy, approved by management, reviewed periodically, communicated and available. It covers principles, scope and commitments.
A.2.3 Alignment with other policies: the AI policy is not an orphan, it aligns with existing security, privacy, quality and ethics policies.

Gap: this is not a code matter. The AI policy is a document that the organisation’s management approves and signs. The blog teaches a neutral, technical editorial stance (no hype, sovereignty, OSS by default under ENS/NIS2), but that is not any specific organisation’s AI policy. Each client has to draft and sign their own.

Minimum template: 1-2 pages with: principles (transparency, human oversight, fairness, accountability, sustainability), scope (which systems), measurable commitments (annual review, impact assessment before deployment, team training), governance (who approves what).

A.3 — Internal organisation (3 controls): GAP

A.3.2 AI roles and responsibilities: roles defined, non-overlapping, communicated. Typically: AI lead, AI risk owner, data steward, AI ethics officer (which may be a single person in small organisations).
A.3.3 Reporting of AI incidents/concerns: a channel for anyone (internal or external) to report a problem with an AI system, with documented follow-up.
A.3.4 Identification of stakeholders: a maintained list of stakeholders (customers, affected persons, regulators, partners) and their expectations.

Gap: also not technical. An organisational decision. The usual way to cover it is to appoint an AI lead (which may be the CIO, the CTO or a new role depending on size), reuse the security reporting channel (typically already in place because of 27001) by extending it to AI, and maintain a live stakeholder register.

A.4 — Resources (6 controls): COVERED

A.4.2 Documented information: documentation of the AIMS.
A.4.3 Data resources: identification and management of the data available for training, evaluation and operation.
A.4.4 Tooling resources: development, validation and monitoring tools.
A.4.5 System resources: hardware, infrastructure, compute.
A.4.6 Human resources: competent personnel.
A.4.7 Financial resources: budget.

Covered by the blog in the three architectural posts:

Anatomy of the stack: seven layers: A.4.5 system resources, A.4.4 tooling.
Seven deployment phases: A.4.5 + A.4.7 (budget implicitly).
Five maturity levels: A.4.5 + A.4.6 (team maturity).
OSS tooling catalogue: A.4.4 tooling.
Data versioning with DVC and lakeFS: A.4.3 data resources.

A.5 — Impact assessment (5 controls): PARTIAL

A.5.2 AI impact assessment process: a documented impact assessment procedure.
A.5.3 Documentation of AI impact assessments: records of the assessments carried out.
A.5.4 Alignment with AI risk treatment: the impact assessment’s decisions feed risk treatment.
A.5.5 Impacts on individuals: specific dimensions covering affected persons (rights, discrimination, privacy).
A.5.6 Societal impacts: dimensions covering society (information, social rights).

Partial: the method exists in the ISO/IEC SC 42 family. ISO/IEC 23894:2023 is the technical standard for AI risk management and NIST AI RMF 1.0 is the freely usable American equivalent. But the organisation has to write its procedure and run it per system before deployment. It is not code, it is discipline.

Minimum template for the impact assessment (3-5 pages per system):

Description of the system (what it does, who it serves, underlying model and stack).
Stakeholders identified.
Potential impacts (intended and unintended) on people, groups and society.
Fairness and robustness metrics applied, with thresholds and results.
Mitigations applied (guardrails, evals, human oversight, rate limiting).
Accepted residual risks, with signed justification.
Review cadence (typically annual, or on substantial change).

A.6 — AI system lifecycle (4 controls): COVERED

A.6.2.2 Objectives for responsible development of AI: responsible development objectives defined per system.
A.6.2.3 Processes for responsible AI design and development: design and development procedures.
A.6.2.4 AI system requirements and specifications: formal specification of the system.
A.6.2.5 Verification and validation: V&V before and during operation.
A.6.2.6 Deployment: deployment procedures.
A.6.2.7 Operation and monitoring: continuous operation and monitoring.
A.6.2.8 Documentation: lifecycle documentation.

Covered by the blog:

The six-stage LLMOps pipeline: the complete master map of the lifecycle.
Anatomy of an LLM request: the forensic version of how it runs in production.
Continuous fine-tuning in production: the A.6.2.3 + A.6.2.5 + A.6.2.6 + A.6.2.7 discipline in real operation.
Evals: the layer after tracing: A.6.2.5 verification and validation.
Retrain: A.6.2.7 operation + continuous iteration.
Modern alignment: DPO, KTO, ORPO, SimPO: A.6.2.3 responsible design.
MLOps landscape 2026: the tooling landscape.

A.7 — Data for AI systems (5 controls): COVERED

A.7.2 Data for development and enhancement of AI: data management policy and procedures for development and improvement.
A.7.3 Acquisition of data: acquisition procedures (origin, authorisation, quality).
A.7.4 Quality of data for AI systems: measurable quality criteria.
A.7.5 Data provenance: data lineage.
A.7.6 Data preparation: preparation procedures (chunking, anonymisation, labelling).

Covered by the blog:

RAG corpus curation: the active librarian: A.7.4 + A.7.5 + A.7.6 in detail (five layers: schema, dedup, PII, anti-contamination, lineage).
Data versioning: DVC and lakeFS: A.7.2 + A.7.5 (the four data artefacts versioned with lineage).
Reranker and hybrid retrieval: A.7.6 preparation + filtering.
LLM Guard: A.7.6 runtime anonymisation with Vault.

A.8 — Information for interested parties (4 controls): COVERED

A.8.2 System documentation and information for users: technical documentation available.
A.8.3 External reporting: the capacity to report to authorities where applicable.
A.8.4 Communication of incidents to users: notification to users when there is an incident.
A.8.5 Information for interested parties: information for other stakeholders.

Covered by the blog:

LLM tracing with OpenTelemetry GenAI: A.8.2 per-request traceability, A.8.3 the capacity to extract forensic reporting.
Prompt versioning with Langfuse and MLflow: A.8.2 documented versioning.
Guardrails and safety in LLMs: A.8.4, gen_ai.guardrail.* spans as the basis for incident notification.
LLM Guard: A.8.4 incident events for retrain.

A.9 — Use of AI systems (3 controls): COVERED

A.9.2 Processes for responsible use of AI: responsible use procedures.
A.9.3 Objectives for responsible use of AI: objectives.
A.9.4 Intended use of AI systems: documentation of the intended use.

Covered by the blog:

Guardrails and safety in LLMs: A.9.2 + A.9.3 (the four lines of defence).
LLM Guard: A.9.2 operational detail.
Evals: the layer after tracing: A.9.3 measurement of objectives.
Retrain: A.9.2 closed loop.

A.10 — Third parties and customer relationships (4 controls): COVERED

A.10.2 Allocation of responsibilities: distribution of responsibilities between AI roles.
A.10.3 Suppliers: procedures for AI suppliers.
A.10.4 Customers: procedures towards customers.
A.10.5 Third parties: procedures for third parties.

Covered by the blog:

The parallel catalogue: OSS vs hyperscalers: A.10.3 supplier assessment with lock-in and contractual sovereignty analysis.
The OSS catalogue for LLMOps: A.10.5 third-party inventory (OSS components with licences and governance).
Anatomy of an LLM request: A.10.2 + A.10.4 in the multi-tenant case.

The roles defined by the standard

ISO/IEC 22989:2022 (AI vocabulary, complementary to 42001) defines six roles. Each organisation has to decide which ones it occupies and document it:

Role	Definition	Main responsibility	Example
AI provider	Organisation that provides the AI system to others	Makes the system available	OpenAI provides GPT-5 via API
AI producer	Organisation that develops the AI system	Design, development, validation	Meta produces Llama 4
AI customer	Organisation that acquires the AI system	Selection, integration, oversight	A consultancy integrating an LLM into its own product
AI partner	Organisation collaborating with another AI role	Shared	A GPU hardware manufacturer
AI subject	Person or group affected by the system	Recipient of the impact	The chatbot’s end user
Relevant authority	Regulator with jurisdiction	External oversight	AEPD, CNMC, EU AI Act authorities

An organisation can occupy several roles at once, which changes the applicable controls. A common pattern in consultancy is: producer + customer + provider towards the end client. The A.10 responsibilities are modulated by the roles.

Example role mapping for the multi-tenant chatbot from the forensic post:

Base model manufacturer (Llama 4): AI producer of the base model.
OSS stack operator (the consultancy): AI producer of the LoRA adapter + AI provider of the chatbot to its clients + AI customer of Meta’s base model.
End client (the insurer): AI customer of the chatbot + AI provider of the customer support service.
The policyholder: AI subject.
AEPD + EU AI Act authority: relevant authority.

Each box in the table generates different obligations. The consultancy, being producer of the adapter, must document A.6 (lifecycle) and A.7 (data) for the adapter. Being provider of the chatbot, it must document A.10.4 (customers). Being customer of the base model, it must document A.10.3 (suppliers) and validate that Meta does its part.

Impact levels and proportionality

42001 does not demand the same rigour of every system. Clause 6.1.2 and control A.5 introduce impact as a modulator. The standard does not define strict categories (unlike the EU AI Act, which does define “prohibited / high risk / limited risk / minimal”), but it recommends using levels according to severity and probability.

Industry practice in 2026 aligns the 42001 levels with the EU AI Act categories:

42001 level	EU AI Act	Examples	Control depth
High	High risk (Annex III)	Credit scoring, HR, health, critical infrastructure	Exhaustive impact assessment, mandatory human oversight, continuous monitoring, adversarial evals, detailed logging, half-yearly management review
Medium	Limited risk	Customer service chatbots that do not automate decisions, productivity assistants	Standard impact assessment, complete guardrails, annual review
Low	Minimal risk	Spam filters, non-personalised content recommendations	Lightweight impact assessment, basic controls

This proportionality is operationally key: implementing 42001 at maximum rigour for a low-risk system is waste; relaxing it on a high-risk one is non-compliance.

The seven minimum AIMS documents

An auditor at Stage 1 (documentation review) asks for between seven and ten documents. The seven essentials:

AI policy (clause 5.2 + A.2.2). 1-2 pages. Approved by management, dated, versioned.
AIMS scope (clause 4.3). List of AI systems in scope, inclusion criteria.
Stakeholder register (clause 4.2 + A.3.4). A maintained list with expectations.
AIMS risk register (clause 6.1). Risks of the management system, not of each model.
Impact assessment procedure (A.5.2) + records of assessments carried out (A.5.3). The procedure plus one or more assessments done.
AI lifecycle procedure (A.6.2), which can literally read “see the six-stage LLMOps pipeline” with references to technical runbooks.
Data management procedure (A.7.2), covering acquisition, quality, provenance, preparation, anonymisation.

Common additional documents:

Responsible use policy (A.9.2) with permitted and non-permitted use types.
AI third-party management procedure (A.10.3, A.10.5) with assessment criteria for AI suppliers.
Internal audit plan + management review agenda (clauses 9.2 + 9.3).

For an organisation with a mature OSS stack, documents 6 and 7 are references to technical artefacts that already exist (pipeline runbooks, DVC configurations, the PII policy in LLM Guard). The real documentary effort sits in documents 1, 2, 3, 4 and 5.

Applied case: the blog’s multi-tenant chatbot → 42001 checklist

We take the system described in the forensic post, a multi-tenant customer support chatbot for insurers on an on-premise OSS stack, and walk it as a 42001 auditor would.

Clause 4 — Context. The AIMS scope includes the chatbot and excludes the internal HR system (a different AI). Stakeholders identified: client insurers, affected policyholders, the AEPD, the EU AI Act authority (once it enters into force on 2 Aug 2026), the supplier Meta (base model), the hardware supplier NVIDIA. → Documented.

Clause 5 — Leadership. AI policy signed by the CEO, in force. Roles assigned: AI lead (CTO), AI risk owner (CISO), data steward (Head of Data), quarterly AI ethics committee. → Documented.

Clause 6 — Planning. AIMS risk register: incomplete documentation, team churn, dependency on a single GPU supplier, EU AI Act regulatory change. AIMS objectives for 2026: 42001 certification before Q4, EU AI Act high-risk compliance before 2 Aug. → Documented.

Clause 7 — Support. Resources: a 4×H100 SXM cluster + the seven stack layers. Competence: 2 MLE + 2 SRE + 1 part-time AI ethics, all with documented training. Communication: AI policy on the intranet + handbook. → Documented.

Clause 8 — Operation. Operating procedures = the six-stage LLMOps pipeline. Impact assessment run before deployment + annual review + review on substantial change (defined as: base model change, major adapter change, expansion to a new tenant). → Documented.

Clause 9 — Evaluation. Monitoring: Langfuse + Tempo + VictoriaMetrics + Grafana. Mandatory dashboard metrics: F1 per guardrail category over real traffic, statistical drift, RAG faithfulness, refusal rate. Quarterly internal audit with written criteria. Half-yearly management review with signed minutes. → Documented.

Clause 10 — Improvement. Incident-driven retrain tickets mapped as nonconformities when severity ≥ HIGH. Documented root cause analysis. Effectiveness verified at the next eval gate. → Documented.

Annex A — By section:

A.2 (Policies): AI policy + responsible use policy. → Documented.
A.3 (Organisation): roles assigned, reporting channel, stakeholder register. → Documented.
A.4 (Resources): seven deployment phases + OSS catalogue + training plan + annual budget. → Documented.
A.5 (Impact): procedure + per-system assessments + fairness metrics applied. → Documented.
A.6 (Lifecycle): LLMOps pipeline + continuous fine-tuning + retrain. → Documented.
A.7 (Data): data versioning + RAG corpus curation + LLM Guard Vault + Presidio. → Documented.
A.8 (Information to parties): OTel tracing + Langfuse + gen_ai.guardrail.* spans + notification to tenants within SLA. → Documented.
A.9 (Use): guardrails + evals + responsible use policy. → Documented.
A.10 (Third parties): OSS vs hyperscalers with lock-in analysis + Meta contract for the base model + contracts with tenants. → Documented.

Result of the walkthrough: certifiable. The typical gaps (A.2.2 written policy, A.3 roles, A.5 impact assessment procedure) are covered as formal documents. The operational clauses (8, 9, 10) lean on the blog’s technical architecture. The distance between “having the architecture” and “having certification” is measured in documentary discipline, not in code.

Cross-mapping with the EU AI Act, NIS2 and ENS

EU AI Act (Reg. 2024/1689) — seven directly aligned articles

EU AI Act article	Topic	Aligned 42001 control	Applies to
Art. 9	Risk management system	A.5 + clause 6	High-risk systems
Art. 10	Data and data governance	A.7 (all)	High-risk systems
Art. 11	Technical documentation	A.6 + A.4.2	High-risk systems
Art. 12	Record-keeping (logs)	A.8.2 + OTel tracing	High-risk systems
Art. 13	Transparency to deployers	A.8.5 + A.10.4	High-risk systems
Art. 14	Human oversight	A.9.2 + documented oversight	High-risk systems
Art. 17	Quality management system	Clauses 4-10	High-risk providers

The main obligations for high-risk systems enter into application on 2 August 2026. Implementing 42001 now builds the management base that deadline demands.

What is missing for EU AI Act compliance that 42001 does not cover:

CE conformity of high-risk systems (declaration of conformity, marking, registration in the EU database).
The specific post-market monitoring required by Art. 72.
Serious incident reporting to authorities within legal deadlines (not only to users).
Transparency obligations towards users for limited-risk systems (Art. 50): chatbots, deepfakes, generated content.
The Art. 5 prohibitions (social scoring, manipulation, real-time biometrics with exceptions).

NIS2 (Dir. 2022/2555) — three pillars with overlap

Asset register (Art. 21.2.f): AI systems within NIS2 scope must be in the asset inventory. → Overlaps with A.4 + clause 4.3 (scope).
Incident notification (Art. 23): significant incidents are notified within 24 h (initial alert) + 72 h (detailed report). → Overlaps with A.3.3 (reporting) + clause 10 (improvement).
Supply chain security (Art. 21.2.d): security assessment of the digital supply chain. → Overlaps with A.10.3 (suppliers).

For essential NIS2 entities that also use AI systems, 42001 covers the AI-specific part that NIS2 requires by inference but does not spell out.

ENS (RD 311/2022)

The Spanish Esquema Nacional de Seguridad already expressly addresses AI in its Annex II (ENS controls). The Basic/Medium/High categories align with the 42001 impact levels. The ENS controls on traceability (op.exp.8), activity logging (op.exp.10) and incident management (op.exp.7) are covered by the same technical artefacts as A.8 and A.5 of 42001. An organisation certified at ENS High category with AI systems is a reasonable effort away from adding 42001.

The five usual certification traps

Trap 1 — Confusing 42001 with EU AI Act compliance. Passing the 42001 audit does not imply conformity with the European Regulation. They are distinct universes with a 60-70% overlap. The trap surfaces when the client asks for CE marking of the high-risk system and the organisation produces only the 42001 certificate.

Trap 2 — Over-documenting. Two-hundred-page manuals with procedures copied from templates, with no measurables and no evidence of application. The auditor asks for the last execution of the procedure; with no records, the procedures are ornament. The practical rule: prefer short documents referencing live technical artefacts to long self-contained ones.

Trap 3 — Under-measuring. Defining AIMS objectives with no operational metrics. “Improve model quality” is a null objective; “F1 per guardrail category ≥ 0.85 over real traffic, measured weekly, reviewed quarterly in the management review” is an auditable objective. The blog has insisted on this in every post on evals, guardrails and retrain.

Trap 4 — Ignoring A.5 until audit day. The impact assessment is the most underrated control and the first one the auditor asks for. Without per-system assessments carried out before deployment, there is no way to demonstrate A.5. The trap surfaces when there is no longer time to produce credible retrospective assessments.

Trap 5 — Assuming 27001 covers the AI part. Organisations that already have 27001 sometimes think “we have half of it done”. That is true for Annex SL (structure) and for A.5/A.6/A.7 of 27001 (not of 42001) as far as infosec goes. It is false for A.5 of 42001 (impact assessment), A.7 of 42001 (AI-specific data quality), A.9 (responsible use) and A.10.4 (AI customers). You have to add, not assume.

What we have not covered (upcoming posts)

Concrete templates of the seven mandatory documents, with drafting examples and metrics. Material for a post along the lines of “The AIMS manual in 7 documents” with a reference frame.
Detailed article-by-article mapping to the EU AI Act with the checklist of technical evidence derivable from the blog’s OSS stack. Especially Arts 11 (technical documentation), 14 (human oversight) and 72 (post-market monitoring).
The combined ENS High category + 42001 case: which ENS controls are covered by which AIMS artefacts, avoiding duplication.
NIST AI RMF 1.0 vs 42001 comparison: many international clients ask for both. How the same artefacts are recycled to satisfy the two frameworks.
42001 for LLM agents and MCP: the new dimensions that emerge when the AI system is agentic (excessive agency, tool use, graduated autonomy). The guardrails post introduced line 3 (tool GR); 42001 has open gaps in this terrain and SC 42 is working on addenda.

References

ISO/IEC 42001:2023 — Information technology — Artificial intelligence — Management system. ISO. https://www.iso.org/standard/81230.html.
ISO/IEC 22989:2022 — Information technology — Artificial intelligence — Artificial intelligence concepts and terminology. Define los roles AI provider/producer/customer/partner/subject.
ISO/IEC 23894:2023 — Information technology — Artificial intelligence — Guidance on risk management. Insumo de A.5.
ISO/IEC 38507:2022 — Governance implications of the use of AI by organizations. Complemento de gobierno.
ISO/IEC 5259 — Data quality for analytics and machine learning (familia). Insumo de A.7.
EU AI Act (Regulation 2024/1689) — texto consolidado en EUR-Lex. Entrada en vigor de obligaciones de alto riesgo: 2 ago 2026.
NIS2 (Directive 2022/2555) — texto consolidado en EUR-Lex.
ENS — Real Decreto 311/2022 — Esquema Nacional de Seguridad, BOE-A-2022-7191.
NIST AI RMF 1.0 (2023) — https://www.nist.gov/itl/ai-risk-management-framework.
EUR-Lex EU AI Act consolidated text — https://eur-lex.europa.eu/eli/reg/2024/1689.
A-LIGN / BSI / Schellman — blogs sobre experiencia de auditoría 42001 con casos reales 2024-2025.

EU AI Act: the technical dossier, article by article, over the blog's on-premise LLM architecture

Mon, 01 Jun 2026 05:30:00 +0200

Sister post to the mapping onto ISO/IEC 42001. That one broke down the AI management system, the certifiable standard. This one breaks down the binding legal regulation that applies without any certification: the EU AI Act is direct law in all 27 Member States, with no transposition and with explicit penalties of up to 35 million euros or 7% of worldwide turnover. The main obligations for high-risk systems enter into force on 2 August 2026; each article applies from its own date, not from the date the organisation gets certified.

TL;DR

Regulation (EU) 2024/1689 (the EU AI Act, “Reglamento Europeo de Inteligencia Artificial”), published in the Official Journal on 12 July 2024, sets obligations by risk tier (prohibited, high, limited, minimal) and by role (provider, deployer, importer, distributor, authorised representative). The obligations for high-risk systems (Annex III: biometrics, critical infrastructure, education, employment, essential public and private services, law enforcement, migration, justice, democratic processes) enter into force on 2 August 2026 and are the category that applies to most LLM projects in a medium-to-large enterprise. This post maps, article by article, the obligations relevant to a high-risk LLM system deployed on-premise: each article states its requirement, identifies which blog post already describes the technical piece that materialises it, and closes with an auditable checklist that a provider hands to a national supervisory authority. The Annex IV technical dossier is reconstructed by pointing its nine mandatory sections at the corresponding technical runbooks. Also covered: the Art. 6 classification and how to decide whether a system falls in as high risk, the Art. 5 prohibitions (what is excluded by construction), the GPAI obligations of Art. 53 that affect anyone building on base models (Llama, Mistral, DeepSeek, Qwen) rather than their own, the full timetable of application dates (5 Aug 2024 entry into force, 2 Feb 2025 prohibitions, 2 Aug 2025 GPAI, 2 Aug 2026 high risk Annex III, 2 Aug 2027 high risk Annex I and systemic GPAI), the Art. 99 penalty table (up to 35 M€ or 7% of worldwide turnover for Art. 5 violations, 15 M€ or 3% for high risk, 7.5 M€ or 1% for incorrect information) and the five frequent compliance traps. The editorial thesis: the technical architecture described in this blog directly covers between 70% and 85% of the Regulation’s technical requirements; the rest is documentary and procedural discipline (FRIA, CE marking, signed declaration of conformity, EU database registration, incident reporting within legal deadlines) built on top of technical artefacts that already exist but running on its own circuit.

The analogy: the type-approval dossier for a new vehicle

A vehicle manufacturer cannot sell a new car in the EU without passing WVTA (Whole Vehicle Type Approval). The process is public and standardised: the manufacturer prepares a technical dossier with dozens of chapters (brakes, emissions, active and passive safety, lighting, noise, weight, dimensions, recyclable materials, driver assistance devices), submits it to a type-approval authority or a notified technical service, which audits the file and, if everything adds up, issues the type approval. The manufacturer then stamps the E mark (E1 Germany, E9 Spain, and so on) and the CE/UNECE plate on every vehicle of that type produced in series. Each vehicle also carries a service book with mandatory inspections, and serious accidents or systemic defects are reported to the competent ministry, which can order recalls.

The EU AI Act adapts exactly this industrial model to high-risk AI software. The provider prepares the Annex IV dossier (nine mandatory sections) with the full technical documentation of the system, runs a conformity assessment (self-assessment for most Annex III cases, notified body for the more sensitive Annex I ones), signs the EU declaration of conformity, registers the system in the European database, and applies the CE marking to the system when it is placed on the market. From there on it must keep a live post-market monitoring system and report serious incidents to the market surveillance authorities of each Member State within legal deadlines ranging from 2 to 15 days depending on severity. Failing to comply, penalties reach 35 million euros or 7% of worldwide turnover depending on which article was breached.

The analogy matters because it bounds expectations: this is not a one-off compliance job, nor a badge you buy. It is an industrial type-approval process, with cadences, evidence, signatures and criminal liability. And most of the technical evidence the dossier asks for already exists in any serious system described in this blog: data lineage, OTel tracing, continuous evals, guardrails, incident-driven retrain, a responsible-use policy. What is missing is assembling it in the correct legal format.

How it fits with ISO 42001, NIS2 and ENS

Before dropping into the articles, a reminder of the editorial position from the post on ISO 42001:

Piece	Nature	Who operates it	Coverage
EU AI Act	Direct EU law	Provider + deployer + national surveillance authorities	AI systems on the EU market, segmented by risk
ISO/IEC 42001	Certifiable management standard	Organisation + certification body	AIMS, organisational governance
NIS2	Transposed cyber directive	Essential/important entities	Asset register, incident notification, supply chain
ENS (Esquema Nacional de Seguridad, Spain’s national security framework, RD 311/2022)	Spanish security regulation	Public sector + its suppliers	Categories B/M/A, certifiable

Implementing 42001 makes it easier to demonstrate articles 9-17 of the EU AI Act, but it is not equivalent to legal compliance. The table of legal obligations and the chain of criminal liability come from the Regulation, not from the standard. An organisation certified against 42001 that deploys a high-risk system without CE marking, without registration in the EU database, without a signed declaration of conformity and without a documented FRIA breaches the Regulation even with the certificate on the wall.

The four risk categories and the Art. 6 classification

The Regulation classifies systems into four risk tiers. The choice of category belongs to the provider and must be documented in the dossier, with technical and legal justification.

Category	Article	Examples	Consequence
Prohibited	Art. 5	Social scoring, manipulation, real-time biometrics with exceptions, indiscriminate facial scraping	Cannot operate in the EU under any circumstances
High risk	Art. 6 + Annex I + Annex III	Credit scoring, HR, education, critical infrastructure, non-real-time biometrics, justice, migration, health	Complies with Arts. 9-17, Annex IV dossier, CE marking, EU DB registration
Limited risk	Art. 50	Chatbots with humans as users, deepfakes, synthetic content	Transparency obligations towards the user
Minimal risk	Everything else	Spam filters, video game NPCs, content suggestions	No specific obligations; a voluntary code of conduct is recommended

The Art. 6 test for deciding whether a system is high risk:

Is it in Annex I? (products regulated by the listed harmonisation legislation: machinery, lifts, toys, medical devices, and so on). If the AI system is a safety component of an Annex I product, it is high risk.
Is it in Annex III? (eight areas: biometrics, critical infrastructure, education, employment, essential public/private services, law enforcement, migration/asylum, justice/democratic processes). If the system falls in any of those areas it is high risk, except if the Art. 6.3 exception is invoked (systems performing a narrow procedural task, improving the result of a previous human activity, detecting decision patterns without influencing the final decision, preparatory tasks).

The Art. 6.3 exception requires formal documentation justifying why it does not apply. In other words, even falling outside is not free: you have to demonstrate why.

For the typical LLM systems covered in this blog:

Customer support chatbot for banking, insurance or healthcare: probably high risk if it automates contractual decisions about the customer, limited risk if it only informs.
Internal HR assistant (CV screening): high risk (Annex III, employment area).
Medical assistant (diagnostic support): high risk (Annex III, healthcare services area).
Fraud detection system: high risk (Annex III, financial services area if it affects access to credit).
Code copilot for developers: minimal risk (does not affect third parties’ fundamental rights).
Internal LLM as a service with no productive use: minimal risk.

The category decision is not an opinion: it is documented and justified in the dossier.

Timetable of application

The obligations arrive in stages. The dates are inflexible:

Date	Applies
1 Aug 2024	The Regulation enters into force
2 Feb 2025	Art. 5 prohibitions + Art. 4 AI literacy obligations
2 Aug 2025	GPAI obligations (Art. 53) + governance + general penalties + designated national authorities
2 Aug 2026	Main high-risk obligations for Annex III + Art. 50 transparency to users
2 Aug 2027	High risk under Annex I (components of regulated products) + GPAI with systemic risk

2 August 2026 is the date that matters to most enterprise LLM projects. We are in June 2026: less than two months to go.

Article-by-article mapping

The sections below follow the structure of the Regulation. For each article the requirement is stated, the blog’s technical artefact that covers it is identified, and it closes with an auditable checklist.

Art. 5 — Prohibited practices (in force since 2 Feb 2025)

What it requires. It prohibits placing on the market, putting into service or using AI systems that:

Manipulate behaviour through subliminal or deceptive techniques that distort decision-making.
Exploit vulnerabilities related to age, disability or socio-economic situation.
Implement social scoring by public authorities.
Perform individual predictive policing based on profiling.
Perform indiscriminate facial scraping to build facial recognition databases.
Infer emotions in workplaces or education, except for medical or safety reasons.
Perform biometric categorisation inferring sensitive attributes (race, political opinion, sexual orientation, and so on).
Real-time remote biometric identification in public spaces by law enforcement, with strict exceptions (terrorism, kidnapping, missing persons) subject to prior judicial authorisation.

The blog’s stack. No piece of the blog facilitates these practices; the OSS catalogue described is oriented towards legitimate tasks. But the provider must explicitly document why the system does not fall under these prohibitions. It cannot be assumed.

Checklist.

Prohibition analysis documented per system, with a written statement of non-applicability.
If the system uses facial biometrics or emotion analysis: specific legal analysis with a formal opinion.
Annual legal review, or on any change of functionality.

Art. 6 — Classification of high-risk systems

What it requires. Determine whether the system is high risk by being in Annex I (safety component of a regulated product) or Annex III (8 areas). If it is in Annex III, assess the Art. 6.3 exception where applicable.

The blog’s stack. The forensic post and the six-stage pipeline describe systems that are typically high risk (multi-tenant chatbot affecting service decisions in a regulated sector).

Checklist.

Art. 6 analysis signed by the organisation’s legal officer.
If Art. 6.3 is invoked: formal documentation of the exception.
Re-assessment if the functionality changes (for example, the assistant moves from informing to deciding).

Art. 9 — Risk management system

What it requires. An iterative system, planned and executed across the whole lifecycle. Identify foreseeable risks, estimate risks under normal use and foreseeable misuse, evaluate emerging risks in post-market monitoring, adopt mitigation measures, communicate residual risks. Effectiveness testing is done under realistic conditions.

The blog’s stack.

The six-stage LLMOps pipeline, the iterative lifecycle.
Evals: the layer after tracing, testing under realistic conditions with golden sets.
Guardrails and safety in LLMs, operational mitigation measures.
Retrain: closing the loop, management of emerging risks via incident-driven retrain.

Checklist.

Risk management document per system, with identification, mitigation, and accepted residual risks signed off.
Periodic review procedure (at least annual, or on substantial change).
Link to the documented improvement loop.

Art. 10 — Data and data governance

What it requires. Training, validation and testing datasets that are relevant, representative, as free of errors and as complete as possible, taking into account the characteristics of the intended purpose. Document:

Data collection and selection.
Processing and annotation.
Biases identified as likely to affect fundamental rights or cause discrimination; measures to prevent them.
Identification of data gaps and how they are addressed.

The blog’s stack.

Data versioning with DVC and lakeFS, the four data artefacts plus end-to-end lineage.
RAG corpus curation: the active librarian, five layers: schema, dedup, PII, anti-contamination, lineage.
LLM Guard: Vault and Anonymize, runtime anonymisation with restitution.

Checklist.

Data governance document per dataset (training / RAG corpus / golden eval / enriched retrain).
Bias analysis by protected category with metrics (parity ratio, equalized odds, calibration).
PII / anonymisation / pseudonymisation procedure with measured F1.
Justification of representativeness for the intended context.
Verifiable chunk→trace lineage.

Art. 11 + Annex IV — Technical documentation

What it requires. A technical dossier with the nine Annex IV sections, drawn up before the system is placed on the market, maintained during operation, and available to authorities for ten years after the last operation. The nine sections:

General description of the system: name, version, purpose, integrator, intended hardware, instructions for use.
Detailed description of design and development: architecture, base models, training methods, design decisions with justification.
Information on monitoring, functioning and control: capabilities, limitations, expected accuracy, behaviour under normal use and foreseeable misuse.
Information on data: datasets used, sources, preparation methods, biases addressed.
Description of the monitoring system and metrics: traces, logs, dashboards.
Description of the QMS and the Art. 17 procedures.
FRIA where applicable (Fundamental Rights Impact Assessment, Art. 27).
Automatically generated logs that the system stores (Art. 12).
EU declaration of conformity under Art. 47, included.

The blog’s stack. It serves as a direct input for each section:

Annex IV section	Technical input from the blog
1. General description	Anatomy of the stack: 7 layers + seven deployment phases
2. Design and development	Six-stage LLMOps pipeline + continuous fine-tuning + modern alignment
3. Capabilities and limitations	Evals + LLM-as-judge
4. Data	Data versioning + RAG corpus curation
5. Monitoring	LLM tracing with OTel GenAI + prompt versioning
6. QMS	Procedures derived from ISO 42001
7. FRIA	Gap, see Art. 27 below
8. Logs	OTel tracing + retention and policies
9. Declaration of conformity	Documentary gap, see Art. 47 below

Checklist.

Annex IV dossier complete, versioned, dated, signed.
10-year retained access guaranteed (immutable / WORM storage).
Update procedure on substantial change.

Art. 12 + Art. 19 — Record-keeping (logs)

What it requires. A high-risk system must be technically capable of generating automatic logs during its operation. Those logs must allow:

Tracing how the system behaved over time.
Facilitating post-market monitoring (Art. 72).
Enabling investigation of serious incidents (Art. 73).
Supporting audits.

The provider must retain the logs for at least six months (or longer if national laws or the QMS require it). For remote biometric identification systems there are additional specific requirements.

The blog’s stack.

LLM tracing with OpenTelemetry GenAI, the canonical substrate. Every request emits a span with trace_id, gen_ai.* attributes, costs, latencies and guardrail decisions.
Prompt versioning: prompt_id + version travel as attributes.
Guardrails and safety in LLMs: gen_ai.guardrail.* attributes record every decision.
LLM Guard: spans per scanner with risk_score and action.
Data versioning: dataset_hash and model_version propagated.

Checklist.

OTel + backend (Tempo, Jaeger) running in production.
Minimum 6-month retention (24-36 months suggested for financial regulation).
WORM / immutable storage.
PII in logs redacted (via LLM Guard Vault or equivalent). Logs are not exempt from the GDPR.
Forensic query procedure with audited permissions.

Art. 13 — Transparency to deployers

What it requires. The provider gives the deployer instructions for use that are clear, complete and accessible, in comprehensible language, covering:

The provider’s identity.
Characteristics, capabilities, limitations (accuracy by category, technical specifications).
Foreseen changes to the system and its metrics.
Human oversight measures (Art. 14).
Expected computational resources and hardware.
When it applies, the expected lifetime of the system and of its maintenance.

The blog’s stack.

The OSS catalogue for LLMOps, which components and with what function.
OSS vs hyperscalers, lock-in analysis and documented dependencies.
Anatomy of an LLM request, capabilities and limitations at forensic level.

Checklist.

User manual in non-technical language for the deployer, plus a detailed technical manual.
Accuracy metrics by category with thresholds.
Change procedure with prior notification.

Art. 14 — Human oversight

What it requires. The system is designed to allow effective human oversight throughout the period of use, with interfaces and procedures that make it possible to:

Understand capabilities and limitations.
Detect malfunctions (automation bias awareness).
Decide not to use the system’s output, to override it, to reverse it.
For remote biometric identification: human verification before acting, by at least two people.

The blog’s stack.

Guardrails and safety, line 3 (Tool GR) with human-in-the-loop for destructive actions.
Evals, metrics on a human-accessible dashboard.
OTel tracing, Langfuse with auditable human sessions.

Checklist.

Oversight interface documented with use cases.
Override / abort capability with no technical restrictions.
Documented training for oversight personnel.
Metrics of the effectiveness of the oversight (override rate, false-negative rate of the oversight itself).

Art. 15 — Accuracy, robustness and cybersecurity

What it requires. High-risk systems are designed and developed to reach an appropriate level of accuracy, robustness and cybersecurity, and to perform consistently throughout their lifecycle. The relevant accuracy metrics are declared in the instructions for use. Resilience against:

Errors, faults and inconsistencies within the environment of use.
Feedback bias during operation (feedback loops).
Attacks that try to exploit the system’s vulnerabilities (data poisoning, model poisoning, model evasion, confidentiality attacks).
Technical and organisational measures to detect, respond to and resolve vulnerabilities.

The blog’s stack.

Quantisation fundamentals, accuracy vs efficiency with reported metrics.
Continuous batching + KV cache, operational robustness.
Guardrails: line 1 input + line 2 retrieval, defence against prompt injection and data poisoning via RAG.
LLM Guard: PromptGuard 2 + injection scanners, direct adversarial mitigation.
Evals: jailbreak resistance + adversarial, robustness metrics evaluated in CI.

Checklist.

Declared accuracy metrics: F1 by category, accuracy, calibration, RAG faithfulness, hallucination rate.
Robustness plan against adversarial inputs (Garak suite / Promptfoo redteam / PyRIT run periodically).
Cybersecurity plan: vulnerability management, patching of the stack (vLLM, its deps, CUDA), secrets rotation.
Analysis of potential feedback loops with drift monitoring.

Art. 17 — Quality management system (QMS)

What it requires. The provider has a written, systematic and proportionate QMS covering (without being limited to):

Regulatory compliance strategy.
Design, verification and quality control of the system.
Testing and validation procedures.
Data management.
The risk management system (Art. 9).
Post-market monitoring (Art. 72).
Incident reporting (Art. 73).
Communication with authorities, deployers and other stakeholders.
Records: documentation, log maintenance.
Resource management.
Accountability: management responsibilities.

The blog’s stack. The QMS is not code, but it leans on code.

ISO/IEC 42001 implemented (previous post) covers practically the whole content of Art. 17.
Six-stage LLMOps pipeline as the reference operating procedure.

Checklist.

QMS manual written, dated, signed, versioned.
Annual internal audit plan with criteria and records.
Management review agenda with signed minutes.
If 42001 is implemented: documented QMS-42001 mapping.

Art. 26 — Deployer obligations

What it requires. Whoever deploys the system (uses it under their own authority, not necessarily the developer) has obligations of their own:

Use the system in accordance with the instructions (Art. 13).
Assign competent, trained human oversight (Art. 14).
Ensure the input data under their control is appropriate.
Monitor operation and notify the provider on detecting problems or serious incidents.
Keep the logs under their control for at least 6 months.
Inform the affected persons when the system is used on them to make decisions (and, in an employment context, also consult their representatives).
For some Annex III cases: complete a FRIA (Art. 27).

The blog’s stack. In the case of the multi-tenant chatbot, the deployer is the client insurer. It must:

Accept the provider’s instructions (the consultancy’s) and sign the terms.
Configure human oversight on its side.
Notify the provider when it detects drift or a serious complaint.
Keep its own logs in addition to the provider’s.

Checklist (for the deployer).

Contract with the provider covering SLAs, responsibilities, exit plan.
Training programme for oversight personnel.
Incident notification procedure towards the provider.
Policy for informing affected persons (in the employment context, notification to representatives).
FRIA where applicable.

Art. 27 — Fundamental Rights Impact Assessment (FRIA)

What it requires. Applicable to deployers that are public bodies or private entities providing public services, or that use systems to assess credit score or life insurance. Before first use, the deployer carries out a FRIA documenting:

A description of the intended use.
Period and frequency of use.
Categories of affected persons.
Specific risks of harm identified.
Human oversight measures.
Mitigation measures should the risks materialise.

The FRIA is notified to the market surveillance authority. Substantial changes force an update.

The blog’s stack. The FRIA is a governance document, not a technical artefact. The blog does not cover it directly. But the technical input is:

The impact analyses from the ISO 42001 post, section A.5 are the natural starting point.
The metrics from the evals post on fairness by category and groundedness feed the risk section of the FRIA.

Checklist.

FRIA procedure documented, aligned with the AIIA in ISO 42005 (published as a complement).
FRIA carried out per system before first use.
Notification to the market surveillance authority.
Periodic review, plus on substantial change.

Art. 47 — EU declaration of conformity

What it requires. The provider draws up a written declaration of conformity per high-risk system, stating:

Identification of the system and the provider.
A statement that it complies with Arts. 8-15 + Art. 17.
Reference to the harmonised standards and common specifications applied.
Where applicable, the notified body and the certificate.
Place and date, signature and name of the authorised signatory.

It must be available to authorities for ten years. It is kept up to date.

The blog’s stack. A pure legal document. Template in Annex V.

Checklist.

Declaration of conformity signed by an authorised person before market placement.
Languages: at least the official EU language of the market where the system is placed.
Re-signing procedure on substantial change.

Art. 48 — CE marking

What it requires. A high-risk system carries the CE marking visibly, legibly and indelibly. For digital systems with no physical part, the marking is included in the documentation or the interface. If a notified body was involved, its number follows.

The blog’s stack. CE marking on a software system typically materialises in:

The product information or “About” page.
The official product documentation delivered to the deployer.
Metadata of the exposed API (custom HTTP header, OpenAPI info).

Checklist.

CE marking visible in the product interface or official documentation.
Notified body number attached where applicable.
Update procedure on substantial change.

Art. 49 — Registration in the European database

What it requires. Before placing on the market or putting into service a high-risk Annex III system (except area 2, critical infrastructure), the provider registers it in the EU database of high-risk AI systems managed by the Commission. Deployers that are public authorities also register their use. Most registration data is public (transparency towards the public).

The blog’s stack. A pure administrative procedure. No technical part beyond having the dossier ready.

Checklist.

Registration completed before first productive use.
Update on substantial change.
Portal access maintained (credentials, contact).

Art. 50 — Transparency to end users

What it requires. Applies to limited risk systems (and, complementarily, to some high-risk ones too):

Chatbots and AI assistants: the person interacting must know they are talking to an AI, unless it is obvious from the context.
Generated synthetic content (text, audio, image, video): marked as AI-generated in the output, in a machine-readable format.
Deepfakes: explicitly declare that the content is artificially generated or manipulated.
Emotion detectors or biometric categorisation: inform the affected persons.

The blog’s stack. Technical materialisation:

A UI banner in the chatbot saying “You are talking to an AI assistant”.
A disclaimer on every exportable response (PDF, email): “Generated by AI”.
Watermarking of the output (perplexity-based, model-fingerprint), optional but useful for deepfakes.
For generated audio, image or video, standard C2PA metadata.

Checklist.

Mandatory UI banner.
Disclaimer on exportable outputs.
If it generates visual content: C2PA marking or equivalent.
Procedure for use of the system on a person without their knowledge (for example, automated CV screening).

Art. 53 — Obligations of GPAI providers (in force since 2 Aug 2025)

What it requires. Providers of GPAI (general-purpose models trained with vast compute, typically foundational: Llama 4, Mistral, DeepSeek, Qwen, Gemma) must:

Maintain technical documentation of the model, accessible to the AI Office and national authorities.
Make information available to downstream providers that will integrate it.
Comply with EU copyright (Art. 4 of Directive 2019/790): an opt-out mechanism for rightsholders.
Publish a summary of the training content (Annex XI, copyright summary).

If the model has systemic risk (10^25 FLOPs threshold, or designated by the Commission), additional obligations apply (Art. 55): model evaluation, adversarial testing, incident reporting, cybersecurity.

The blog’s stack. For an organisation that uses GPAI models (Llama 4, Mistral) and does not train them from scratch:

It is not a GPAI provider; it is a downstream provider integrating GPAI into its system.
It must hold the GPAI technical documentation (Llama paper, Mistral docs) and reference it in its own Annex IV documentation.
Analysis of the specific GPAI licence (Llama Community License, Apache 2.0, and so on).
The post on modern alignment and continuous fine-tuning describe how a LoRA adapter on top of the GPAI does not turn the downstream party into a GPAI provider, as long as it does not train a new model from scratch.

Checklist (downstream provider).

Inventory of GPAI models used, with version, source, licence, referenced documentation.
Mapping of responsibilities between the upstream GPAI provider and us as downstream.

Art. 72 — Post-market monitoring

What it requires. A documented post-market monitoring system proportionate to the risk, collecting data on performance throughout the lifecycle, including interaction with other AI systems. It allows the provider to:

Assess continuous compliance with Arts. 8-15.
Adopt the necessary corrective measures.
Detect trends in real use (drift, abuse).

The monitoring plan is part of Annex IV and is maintained throughout the system’s life. The Commission publishes a template in 2026.

The blog’s stack.

OTel tracing + Langfuse, the base.
Continuous evals, operational metrics as online gates.
LLM-as-judge, judges over production sampling to detect degradation.
Incident-driven retrain, the closed improvement loop.

Checklist.

Post-market monitoring plan documented per system.
Operational metrics defined with thresholds and periodic review.
Corrective action procedure on alert.
Integration with Art. 73 (when an alert equals a serious incident).

Art. 73 — Serious incident reporting

What it requires. The definition of a serious incident (Art. 3(49)):

Death or serious harm to health.
Serious disruption of critical infrastructure.
Infringement of legal obligations intended to protect fundamental rights.
Serious damage to property or the environment.

The provider reports to the market surveillance authority of the Member State where the incident occurred. Deadlines:

Type of incident	Maximum deadline from becoming aware
General	15 days
Death	10 days
Critical infra affected, or widespread infringement	2 days

The first report may be preliminary; the complete one follows later. Internal investigation is mandatory, as is cooperation with the authorities. Corrective action is proportionate.

The blog’s stack. The technical flow:

OTel tracing provides the traceability for forensic investigation.
Guardrails and safety emits the canonical incident_event (category, severity, trace_id).
Incident-driven retrain materialises the corrective action.

Checklist.

Incident reporting procedure documented with deadlines and templates.
Designated person (typically DPO + AI Risk Owner) with responsibility and training.
Annual dry-run of the procedure (drill).
Technical integration between the guardrails / tracing layer and the notification channel.

The assembled Annex IV dossier: SVG of the complete file

The diagram shows the blog’s editorial asymmetry: seven of the nine Annex IV sections are covered directly by the technical stack. Only the FRIA (section 7) and the signed declaration of conformity (section 9, not shown in the SVG because it is a one-page document) are documentary gaps needing express administrative work. The main compliance effort is assembling and signing, not building from scratch.

Applied case: the multi-tenant chatbot assessed against the AI Act

We take the system from the forensic post, a multi-tenant customer support chatbot for insurers, and walk through it as the provider of the system.

Art. 6 classification: the chatbot helps customers understand products, check status and open tickets. It does not automate contractual decisions (it does not approve claims, it does not calculate premiums). Does it fall into Annex III area 8 (essential private services)? That depends on the use. If the insurer uses it only for informational support, it is limited risk (Art. 50 applies). If it uses it to assess claim declarations, it is high risk. Documenting the analysis is mandatory.

We assume high risk for the complete walkthrough:

Art. 5 prohibitions: written statement of non-applicability. ✓
Art. 9 risk management: per-system document with identified risks (hallucination, dialect bias, PII leakage, jailbreak), applied mitigations (RAG over a curated corpus, guardrails with 4 lines, LLM Guard Vault, continuous evals) and signed accepted residuals. ✓ (input: pipeline + guardrails + evals).
Art. 10 data governance: governance of the four datasets (training adapter, insurer RAG corpus, golden eval, enriched retrain) with biases analysed, PII anonymised, lineage in place. ✓ (input: data-versioning + rag-corpus-curation + LLM Guard).
Art. 11 + Annex IV: dossier with 9 sections drafted, signed, accessible for 10 years in a WORM bucket. ✓ (7 sections from the blog, 2 new).
Art. 12 + Art. 19 logs: OTel + Tempo + Langfuse with 24-month retention, PII redacted by Vault. ✓
Art. 13 transparency to deployers: user manual for the insurer plus a technical manual. ✓ (input: OSS catalogue + request anatomy).
Art. 14 human oversight: Langfuse + Grafana dashboard, human escalation protocol for critical cases, training for the insurer’s staff. ✓
Art. 15 accuracy, robustness, cybersecurity: F1 by category declared, Promptfoo redteam adversarial suite run monthly, stack cybersecurity plan (vLLM patching, secrets rotation). ✓
Art. 17 QMS: ISO 42001 implemented and certified. ✓
Art. 26 deployer: the contract with the insurer includes the deployer’s obligations. ✓
Art. 27 FRIA: the insurer carries out a FRIA before first use (it is a private essential-service entity). ✓ (deployer’s responsibility, the provider assists).
Art. 47 declaration of conformity: signed by the consultancy’s CTO before market placement. ✓
Art. 48 CE marking: visible in the chatbot interface and in the official documentation. ✓
Art. 49 EU DB registration: completed before first productive use. ✓
Art. 50 transparency to users: UI banner “You are talking to an AI assistant”, disclaimer on exportable responses. ✓
Art. 53 GPAI: Llama 4 documentation (the base model) referenced, plus EU copyright analysis and responsibility mapping. ✓
Art. 72 post-market monitoring: documented plan, OTel + continuous evals + retrain already running. ✓
Art. 73 incident reporting: procedure, designated owner, annual dry-run. ✓

Result: certifiable and deployable on the EU market on 2 August 2026. The key gaps (FRIA, CE marking, EU DB registration, declaration of conformity) are documentary work over technical artefacts that already exist, not new technical projects.

Penalties (Art. 99)

The penalty table is proportional to worldwide turnover or to an absolute cap, whichever is higher:

Violation	Penalty cap
Art. 5 (prohibited practices)	Up to 35 M€ or 7% of annual worldwide turnover
Other obligations (Arts. 8-22, 26-50, 72-73, and so on)	Up to 15 M€ or 3% of annual worldwide turnover
Incorrect or misleading information to authorities	Up to 7.5 M€ or 1% of annual worldwide turnover

For SMEs and startups the caps are the lower of the two figures (not the higher). The proportionality mitigation exists but requires formal demonstration.

In addition, national authorities can order:

Immediate suspension of the system on the market.
A mandatory recall.
Mandatory public communication of the penalty.

The five frequent compliance traps

Trap 1 — Assuming the GPAI model used already covers the obligations. The downstream provider remains responsible for the integrated system: neither Meta nor Mistral nor DeepSeek takes on the obligations of whoever builds on their models. The trap surfaces when the authority asks for the dossier and the team points at the Llama model card as if that were enough.

Trap 2 — Confusing ISO 42001 with EU AI Act conformity. Holding a 42001 certificate does not imply conformity: certification is not a FRIA, not CE marking, not EU database registration, not a signed declaration of conformity. Standardisation is progressing, but until ISO 42001 is published as a harmonised standard (it was not at the end of 2025), there is no presumption of conformity. The table of legal obligations runs on its own circuit.

Trap 3 — Forgetting incident reporting within the legal deadline. Fifteen days sounds long until an incident lands in the middle of the summer holidays. Two days for critical infrastructure is not negotiable. Without a documented procedure and an annual drill, the deadline is missed in silence. A penalty for misleading information (1% of worldwide turnover) follows if the incomplete reporting is discovered.

Trap 4 — Underestimating the deployer. The AI Act assigns obligations to both the provider and the deployer. A company that uses a hosted LLM integrated into its service (without developing it) is still a deployer with obligations of its own (Art. 26): human oversight, FRIA where applicable, notification to affected persons. The trap surfaces when the deployer assumes that “responsibility lies with the model provider”. It does not, not entirely.

Trap 5 — Leaving the FRIA until last. The FRIA (Art. 27) requires a fundamental rights impact analysis with qualitative dimensions (discrimination, privacy, social rights). It is not an afternoon’s document. It is carried out before first use, not after a problem is detected. Public deployers and private essential-service providers who leave it for “whenever they ask” take 4-8 weeks to produce a credible one, time they typically do not have when the inspection arrives.

What we have not covered (upcoming posts)

Concrete templates for each document: EU declaration of conformity (Annex V), full Annex IV dossier section by section, FRIA, post-market monitoring plan, initial serious incident report. Material for a post along the lines of “The EU AI Act compliance folder in 12 templates”.
Voluntary codes of practice published by the Commission under Art. 56, useful for limited-risk systems that want to demonstrate good behaviour without a legal obligation.
Comparative analysis of notified bodies for systems requiring third-party conformity assessment (mainly Annex I).
How compliance changes for LLM agents, systems with graduated autonomy, tool calling and the capacity to act. SC 42 and the AI Office are working on specific guidance.
The complete map of national authorities designated under Art. 70, plus the sectoral authorities that retain jurisdiction (AEPD for privacy, CNMV for financial services, Banco de España for banking, AESA for aviation).

References

Regulation (EU) 2024/1689 (EU AI Act) — Texto consolidado en EUR-Lex: https://eur-lex.europa.eu/eli/reg/2024/1689. Diario Oficial L 1689/12.7.2024.
EU AI Act Explorer (AI Act Service Desk, Comisión Europea): https://ai-act-service-desk.ec.europa.eu/en/ai-act-explorer.
AI Act Text portal (artificialintelligenceact.eu): artículos individuales con anotaciones.
Anexo IV — Technical documentation: estructura de los nueve apartados obligatorios.
Anexo V — EU declaration of conformity: plantilla obligatoria.
Anexo III — High-risk AI systems: las ocho áreas que clasifican un sistema como alto riesgo.
Anexo XI — Copyright training summary template: para GPAI providers.
NIST AI RMF 1.0 (2023) — https://www.nist.gov/itl/ai-risk-management-framework.
ISO/IEC 42001:2023 — sistema de gestión, complemento facilitador.
ISO/IEC 42005 — Impact assessment AI (publicada 2025 como guía técnica para FRIA).
Draft Commission guidance on serious incident reporting (2025) — borrador en consulta para Art. 73.

The parallel catalogue: the six LLMOps stages in open source and on the hyperscalers (AWS, GCP, Azure)

Sat, 23 May 2026 07:00:00 +0200

TL;DR

The previous forensic post used a single request to walk the six stages of the LLMOps pipeline and the two cross-cutting components. This post walks the same stages but crosses them with three extra columns: how each stage is built in open source on-premise, and what the equivalent services are on AWS, GCP and Azure. It is not a migration guide nor a cost benchmark: it is a catalogue of equivalences with its gaps. The general pattern you will see: OSS gives you control, sovereignty and free composition in exchange for expensive operations; the hyperscalers give you integration and time-to-market in exchange for lock-in on margins, data contracts and political dependency. For scenarios subject to ENS / NIS2 with critical customer data, OSS wins by default; for discovery projects where time-to-market is the deciding metric, the hyperscaler wins by default. The interesting part is in the middle. As a concrete thread, at the end we take the multi-tenant chatbot from the previous post and port it to AWS piece by piece to show what disappears, what appears, and where the lock-in materialises.

You are here: the same six stages, but by column

This post shares a map with the previous one, the six stages and the two cross-cutting pieces are all active, but it changes the cut: instead of following a request horizontally, it makes the vertical cut and shows which tools live in each stage depending on the deployment model.

The analogy: your own bakery and the franchise

A baker opens a business. There are two possible models.

He can open his own bakery: rent the premises, buy the oven, choose the flour suppliers, hire his master baker, write his recipes, set the prices, dress the shop window. The day he wants to launch an organic rye sourdough loaf, he asks nobody for permission. The day the price of flour goes up, he finds another supplier. But he pays for all of it: the initial investment, the risk, the daily operations, the months when he misreads the neighbourhood. The bakery is his.

Or he can join a franchise: the franchisor hands over the turnkey premises, the oven with a maintenance contract, the suppliers already negotiated, the operating manuals, the written recipes, the centralised marketing, the loyalty app, the POS system. The learning curve is weeks, not years. But the recipes belong to the franchisor, so do the suppliers, the price of bread is in the catalogue, and the day the croissant formula changes he gets an informational email, not a business decision.

Both bakeries turn out bread. Both meet health regulations and generate income. The operational difference is enormous and it is not about technology: it is about ownership, control and timescale.

The parallel with LLMOps is direct. The on-premise OSS stack is the bakery you own. The managed stack on the hyperscalers is the franchise. The pieces that appear in each stage are functionally equivalent, at the end of the day both solve the same technical problem, but the governance model, the operational cost, the lock-in and the compliance guarantees are different. This post builds the parallel catalogue so that the choice is not made by default.

Quick recap of the previous post

In the forensic post we followed one specific request: a premium-es user of an insurance company asking “How do I cancel my premium subscription?” to the multi-tenant support chatbot of the SaaS provider that hosts it. The route crossed the six stages of the LLMOps pipeline, Data, Tune, Eval, Deploy, Observe, Retrain, plus the two cross-cutting components, prompt versioning and data versioning, over an on-premise infrastructure: RKE2 with Cilium BGP, a 4×H100 SXM cluster, an RTX 4090 for development, vLLM on Kubernetes, Langfuse + OTel + Prometheus + Tempo, Postgres + Qdrant, DVC + lakeFS + MinIO, Kafka and MLflow. The system meets ENS / NIS2 and keeps trace_id propagated end to end.

What comes now is that same system, piece by piece, showing for each box which tool does the job if you are on public cloud, because the integrator’s question is rarely “OSS yes or no?”: it is “what do I lose and what do I gain if I take this box managed?”. And the answer differs box by box.

Stage 1 — Data

The problem. There are three sub-problems the Data stage solves, frequently confused with one another. First, versioning and identity of the corpus and of the training datasets (making sure a dataset_id, dataset_version exists and propagates). Second, storage and serving of the operational corpus (object store + vector index + structured text). Third, streams and ingestion from source systems with CDC, transformation and stable schemas (Schema Registry).

Reference OSS stack. Versioning lives in DVC (pointers in git, content in an object store) combined with lakeFS for branch/merge semantics over data. The post on data versioning goes into the functional difference. The object store is MinIO or Ceph. The vector index is Qdrant or Milvus for large corpora (millions of chunks) and pgvector on Postgres 18 for small cases where having one component fewer pays off operationally. The stream layer is Kafka (pure Apache or Redpanda) with a Schema Registry (Confluent or Karapace OSS), CDC with Debezium or Flink CDC, transformation with Flink or Spark Structured Streaming. The catalogue / lineage is DataHub, Apache Atlas or OpenMetadata with OpenLineage events between systems. The post on PostgreSQL + Qdrant ingestion and the post on RAG over Kafka cover the detailed operations.

Hyperscaler equivalents. On AWS, the corpus lives in S3 (with versioning enabled, which is the cheap substitute for serious data versioning), tabular queries in Athena or Redshift, the vector index in Amazon OpenSearch with the vector plug-in or in Amazon Aurora pgvector. The stream layer is MSK (managed Kafka) or Kinesis Data Streams, CDC with AWS DMS, transformation with Glue Streaming or MSK Connect. The catalogue is AWS Glue Data Catalog + AWS Lake Formation for data governance. And for the RAG case there is also Amazon Bedrock Knowledge Bases, the managed shortcut: you give it S3, it indexes into OpenSearch or Aurora pgvector, exposes a retrieval API and the operations end there, in exchange for paying per indexed chunk and per query.

On GCP, the corpus lives in Cloud Storage (with object versioning), the analytical warehouse is BigQuery (with BigQuery Vector Search already integrated), the dedicated vector store is Vertex AI Vector Search (formerly Matching Engine). The stream layer is Pub/Sub + Dataflow, CDC with Datastream. Catalogue and lineage are Dataplex (which in 2024-2025 absorbed Data Catalog and added automatic lineage). The managed equivalent of Knowledge Bases is Vertex AI Search (formerly Discovery Engine).

On Azure, the corpus lives in ADLS Gen2, tabular queries in Microsoft Fabric / Azure Synapse, the vector index in Azure AI Search (vector mode) or Azure Cosmos DB for PostgreSQL with pgvector. The stream layer is Event Hubs + Stream Analytics or Microsoft Fabric Real-Time Intelligence, CDC with Azure Data Factory. The catalogue is Microsoft Purview, which covers catalogue, lineage and data governance integrated with Entra ID.

Summary table — Data stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Object store	MinIO, Ceph	S3	Cloud Storage	ADLS Gen2
Dataset versioning	DVC, lakeFS	S3 Versioning (limited), Lake Formation	GCS Versioning, Dataplex	ADLS versioning, Purview
Vector index	Qdrant, Milvus, pgvector	OpenSearch, Aurora pgvector, Bedrock KB	Vertex Vector Search, BigQuery VS	Azure AI Search, Cosmos pgvector
Stream + CDC	Kafka + Debezium + Flink	MSK / Kinesis + DMS + Glue	Pub/Sub + Datastream + Dataflow	Event Hubs + ADF
Schema Registry	Karapace, Confluent OSS	Glue Schema Registry	Pub/Sub schemas	Schema Registry (Event Hubs)
Catalogue + lineage	DataHub, Atlas, OpenLineage	Glue Catalog + Lake Formation	Dataplex	Purview
Managed end-to-end RAG	— (you build it)	Bedrock Knowledge Bases	Vertex AI Search	Azure AI Studio Knowledge

Where the names mislead. S3 Versioning is not DVC. It keeps object versions but has no notion of a dataset (which objects together make up version 3 of the enriched set?), does not propagate dataset_hash to the trainer, does not integrate with experiment tracking, and does not fail a CI run if a dataset breaks schema. Covering that properly on AWS requires combining it with Lake Formation, Glue Data Catalog and your own records in SageMaker Experiments. The same applies on GCP with Dataplex and on Azure with Purview. The gap is real and is paid for in operations or in broken lineage.

Stage 2 — Tune

The problem. Producing a new model_id, model_version, typically a LoRA adapter over a stable base, as the post on continuous fine-tuning explains, with lineage back to the dataset that trained it and experiment tracking that lets you reproduce it six months later.

OSS stack. Technical core: HuggingFace Transformers + PEFT (LoRA, QLoRA), bitsandbytes for quantisation, DeepSpeed or FSDP for parallelism. Experiment tracking: MLflow (authoritative) or Weights & Biases self-hosted. Convenience frameworks: Axolotl and Llama Factory wrap the machinery above in declarative configuration. Distributed orchestration: Kubeflow Training Operator or Ray Train. On small infrastructures, direct scripts with Slurm or K8s Jobs over GPU pools. The lineage chain dataset → run → model is closed by registering the dataset as an MLflow input artefact.

Hyperscaler equivalents. On AWS, SageMaker Training Jobs covers most workloads, SageMaker HyperPod covers large training runs with resilience to node failures, and SageMaker JumpStart offers click-to-train fine-tuning over a catalogue of pre-curated models. For fine-tuning Bedrock models (hosted Claude, Llama, Mistral) there is Bedrock Custom Models: you upload the dataset to S3, Bedrock trains, and it returns a private endpoint with provisioned throughput. The equivalent experiment tracking is SageMaker Experiments or managed MLflow in SageMaker (yes, AWS has officially hosted MLflow since 2024).

On GCP, Vertex AI Custom Training runs any container with GPUs or TPUs; Vertex AI Tuning is the managed API for fine-tuning Gemini and Model Garden models. Experiment tracking in Vertex AI Experiments (with MLflow compatibility).

On Azure, Azure ML Training Jobs on your own clusters or managed compute; Azure OpenAI fine-tuning for fine-tuning GPT and the o-series; Azure ML Experiments with MLflow natively integrated since 2022.

Summary table — Tune stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Training framework	HF Transformers + PEFT	SageMaker SDK	Vertex AI SDK	Azure ML SDK
Quantisation / parallelism	bitsandbytes, DeepSpeed, FSDP	SageMaker libs + HF support	Vertex + HF support	Azure ML + HF support
Managed fine-tuning (black box)	—	Bedrock Custom Models, JumpStart	Vertex Tuning (Gemini)	Azure OpenAI fine-tuning
Distributed on a cluster	Kubeflow, Ray Train, Slurm	SageMaker HyperPod	Vertex AI Training (multi-node)	Azure ML compute clusters
Experiment tracking	MLflow, W&B self-hosted	SageMaker Experiments, managed MLflow	Vertex Experiments	Azure ML + MLflow
Access to the base model	Whatever you download (Llama, Mistral, Qwen)	Bedrock catalog + HF Hub	Vertex Model Garden + HF Hub	Azure ML model catalog + HF Hub

Where the names mislead. The managed fine-tunings (Bedrock Custom, Vertex Tuning, AOAI fine-tuning) are a black box: you do not choose hyperparameters beyond a handful, you do not see the trainer’s detailed logs, and you cannot inspect the dataset once it is inside their pipeline. The experiment tracking they offer is not comparable to MLflow sitting next to the trainer, where you can capture any metric and artefact. For scenarios where operationally you do not need inspection this is liberating; for ENS / NIS2 scenarios where you have to prove what trained what, the black box fails by construction.

Stage 3 — Eval

The problem. Validating candidates before and after promotion against a golden set, with operational metrics (faithfulness to the RAG, tone, format compliance, toxicity, jailbreak resistance, PII leakage) run as gates in CI and as online sampling. Covered in the post on evals and the one on guardrails.

OSS stack. Eval suites: DeepEval, RAGAS (specialised in RAG), Promptfoo (declarative, ideal for CI), lm-eval-harness (academic), HELM. Evals integrated with tracing: Langfuse Evals, Phoenix Arize OSS. LLM-as-judge judges: any local OSS model; in serious systems, two different judges to reduce bias. Safety and guardrails: NeMo Guardrails (NVIDIA), Guardrails AI, LlamaGuard + PromptGuard (Meta), ShieldGemma (Google, open weights), PII detectors such as Presidio (Microsoft) on-prem.

Hyperscaler equivalents. On AWS, Bedrock Model Evaluation offers automatic evals (toxicity, accuracy, robustness) and human-in-the-loop, Bedrock Guardrails covers the safety layer (denied topics, PII, prompt injection, contextual grounding check), and SageMaker Clarify adds bias and explainability over general models.

On GCP, Vertex AI Evaluation Service runs evals with automatic metrics and an LLM judge, Vertex AI Model Armor and the safety filters built into the Gemini API cover the guardrails layer. Vertex AI Studio exposes interactive Eval for prompt iteration.

On Azure, the Azure AI Evaluation SDK runs offline evals against datasets, Azure AI Content Safety covers safety (Prompt Shields against jailbreak, Groundedness detection, content categories, PII detection). All accessible from Azure AI Foundry.

Summary table — Eval stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Automatic eval suite	DeepEval, RAGAS, Promptfoo	Bedrock Model Evaluation	Vertex AI Evaluation Service	Azure AI Evaluation SDK
LLM-as-judge	Any OSS model	Bedrock judge models	Vertex judge (Gemini)	Azure OpenAI judges
Golden set management	Langfuse datasets, manual	SageMaker Ground Truth datasets	Vertex Datasets	Azure ML Datasets
Guardrails (jailbreak, PII, prompt injection)	NeMo Guardrails, LlamaGuard, Presidio	Bedrock Guardrails	Vertex Model Armor + Gemini safety	Azure AI Content Safety (Prompt Shields, Groundedness)
Eval in CI	Promptfoo + GitHub Actions	Bedrock Eval API + CodeBuild	Vertex Eval API + Cloud Build	Azure AI Eval + Azure Pipelines

Where the names mislead. Managed guardrails are convenient but opaque: the Bedrock Guardrails rules are configurable but the detection implementation cannot be inspected; the same goes for Azure AI Content Safety. In OSS, NeMo Guardrails shows you the Colang graph and Presidio shows you the recognisers, auditable and modifiable. For regulated systems where an auditor asks “how exactly does it detect PII?”, OSS answers with code; the cloud answers with documentation.

Stage 4 — Deploy

The problem. Serving tokens to the end user with predictable latency and throughput, a decent cost / token ratio, hot-swap adapter support, and multi-tenancy if the business demands it. Covered in the posts on KV cache, PagedAttention, disaggregated serving, vLLM on K8s, LLM operators and multi-tenant cluster.

OSS stack. Inference engine: vLLM (PagedAttention, prefix caching, LoRA hot-swap, OpenAI-compatible API) as the reference, TensorRT-LLM for maximum optimisation on Hopper / Ada, SGLang for workloads with heavy prompt restructuring, TGI (Hugging Face) as a mature alternative, llama.cpp for edge and CPUs, NVIDIA Dynamo for multi-node disaggregated serving on large clusters. Orchestration on Kubernetes: KServe, KubeRay, dedicated operators such as llm-d, vLLM Production Stack and KAITO. Gateway / control plane: Envoy AI Gateway, LiteLLM Proxy, Portkey AI Gateway, Kong AI Gateway. Triton Inference Server covers mixed workloads (LLM + traditional) where a single backend matters.

Hyperscaler equivalents. On AWS, two different routes. The managed-by-model route is Amazon Bedrock: a catalogue of hosted models (Claude, Llama, Mistral, Cohere, Titan), pay per token or Provisioned Throughput with an SLA, Bedrock Prompt Caching as the conceptual equivalent of vLLM’s prefix caching, and integrated Bedrock Agents and Bedrock Knowledge Bases. The managed-by-infrastructure route is SageMaker Endpoints (real-time, async, serverless, batch) with Inference Components to densify multiple models on one instance. Own hardware: AWS Inferentia and Trainium via the Neuron chip, an alternative to NVIDIA with a better cost / token on stable workloads if you compile your model.

On GCP, Vertex AI Prediction Endpoints runs your containers or models from the Model Garden, the Gemini API via Vertex AI offers the managed Gemini models, and Cloud TPU v5e / v5p / Trillium (v6) is the in-house hardware competing with the H100 for training and inference. For sovereignty there is Google Distributed Cloud air-gapped, which brings Vertex AI to a certifiable on-premise rack.

On Azure, Azure OpenAI Service serves OpenAI models (GPT-4.1, o-series, GPT-image), Azure ML Managed Online Endpoints runs any model (OSS included, via container), and Azure AI Foundry models absorbed in 2025 the catalogue of open models served as-a-service. Hardware: Azure ND H100 v5, ND H200 v5, ND GB200 v6 and the in-house bet Microsoft Maia 100 for internal inference.

Summary table — Deploy stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Inference engine	vLLM, TensorRT-LLM, SGLang, TGI	Bedrock (managed model), SM Endpoints (your container)	Vertex Prediction, Gemini API	Azure OpenAI, Azure ML Endpoints
Prefix / prompt caching	vLLM native	Bedrock Prompt Caching	Vertex AI context caching	Azure OpenAI prompt caching
Adapter hot-swap (LoRA)	vLLM `--enable-lora`, S-LoRA	Bedrock Custom Models endpoints	Vertex Tuning endpoints	Azure OpenAI fine-tuned deployments
Disaggregated serving	NVIDIA Dynamo, vLLM PD-disagg	— (managed internally, not exposed)	— (managed internally, not exposed)	— (managed internally, not exposed)
Accelerator hardware	NVIDIA H100/H200/B200, AMD MI300	Inferentia, Trainium, NVIDIA	TPU v5/v6, NVIDIA	Maia, NVIDIA
AI Gateway / proxy	Envoy AI Gateway, LiteLLM, Portkey, Kong	API Gateway + Bedrock	Vertex AI + Apigee	Azure API Management + AOAI
K8s orchestration	KServe, KubeRay, llm-d, KAITO	EKS + SageMaker Operators	GKE + Vertex AI	AKS + KAITO

Where the names mislead. Bedrock Prompt Caching and Vertex context caching sound equivalent to vLLM’s prefix caching, but operationally they are different: the cache lives on the hyperscaler’s plane, its eviction policy is opaque, its cost is billed separately, and you cannot easily see the hit ratio per tenant. In vLLM you see the hit ratio in Prometheus metrics and you decide the policy. The same applies to disaggregated serving: the hyperscalers implement it internally to cut their own cost of serving, but they do not expose prefill/decode control to the user. If you need your workload’s TTFT to be controlled separately from TPS, that lever is not available.

Stage 5 — Observe

The problem. End-to-end LLM traces with trace_id propagated through every component, runtime metrics per tenant, online scoring (LLM judge over sampling), statistical drift, and safety / guardrails monitoring. Covered in the posts on AgentSight tracing, MCP observability with OTel and eBPF + drift.

OSS stack. Base standard: OpenTelemetry (specification + collector + SDKs) with the gen_ai semantic conventions that stabilised in 2025. Backends: Tempo or Jaeger for traces, Prometheus for metrics, Loki for logs, Grafana as the common UI. LLM-specific layer: Langfuse (self-hosted with an optional EE licence) and Phoenix Arize OSS. eBPF layer for low-level observability: Pixie, Hubble, and Cilium Tetragon for runtime security. Drift: Evidently AI, NannyML, Alibi Detect.

Hyperscaler equivalents. On AWS, CloudWatch (metrics + logs) + AWS X-Ray (traces) form the base, CloudWatch Application Signals adds OTel-compatible APM, and Amazon Managed Prometheus and Amazon Managed Grafana serve the plane if you want to keep Prom + Grafana without operating them. Bedrock logging integrates with CloudWatch and S3. ADOT (AWS Distro for OpenTelemetry) is the official collector.

On GCP, Cloud Monitoring + Cloud Logging + Cloud Trace + Cloud Profiler form the quintet, all OTel compatible. Vertex AI Model Monitoring offers drift detection (feature skew, prediction drift) integrated with runs.

On Azure, Azure Monitor + Application Insights + Log Analytics cover the APM stack with native OTel, Azure ML Model Monitor adds drift and data quality, and Azure OpenAI diagnostic logs enrich the traces with token and model metadata.

Summary table — Observe stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Traces (OTel)	OTel + Tempo / Jaeger	X-Ray + ADOT, App Signals	Cloud Trace	App Insights + Azure Monitor
Metrics	Prometheus + Grafana	CloudWatch + AMP / AMG	Cloud Monitoring	Azure Monitor Metrics
Logs	Loki, ELK	CloudWatch Logs	Cloud Logging	Log Analytics
LLM-specific (prompt, scores, sessions)	Langfuse, Phoenix Arize OSS	Bedrock logging + CW + custom	Vertex AI tracing + custom	App Insights + AOAI logs + custom
Drift detection	Evidently, NannyML, Alibi Detect	SageMaker Model Monitor	Vertex AI Model Monitoring	Azure ML Model Monitor
eBPF / runtime	Pixie, Hubble, Tetragon	— (no direct equivalent)	GKE Dataplane v2 / Cloud Service Mesh	Azure CNI + Defender for Cloud

Where the names mislead. The classic cloud APM tools (X-Ray, Cloud Trace, App Insights) do not understand prompt versioning or adapter id as native concepts. They accept the gen_ai.* attributes as dimensions, but the UIs do not prioritise those views. Langfuse and Phoenix do, because they are designed for LLM. In cloud, the usual pattern is to send dual: APM to the managed service for infrastructure + self-hosted Langfuse / Phoenix for the LLM plane. That pays off.

Stage 6 — Retrain + cross-cutting pieces

The problem (Retrain). Closing the loop feedback → triage → enriched dataset → new adapter, with a mixed cadence (quarterly + incident-driven). Covered in the Retrain post.

OSS Retrain stack. Orchestration: Apache Airflow, Prefect, Dagster or Argo Workflows and Kubeflow Pipelines for K8s-native. Feature store where applicable: Feast. Annotation and human-in-the-loop: Argilla, Label Studio, Trubrics. Structured feedback capture: your own Postgres table + Langfuse scores + Phoenix annotations. Closed-loop lineage: OpenLineage tying dataset → run → model → deployment → feedback → next dataset.

Hyperscaler Retrain equivalents. On AWS, SageMaker Pipelines orchestrates the cycle, SageMaker Ground Truth and A2I (Augmented AI) manage annotation and HiL, and SageMaker Model Monitor fires alerts that can invoke retrain pipelines. AWS Step Functions serves as a more general alternative orchestrator.

On GCP, Vertex AI Pipelines (based on Kubeflow Pipelines, compatible) orchestrates, Vertex AI Data Labeling Service annotates, Vertex AI Feature Store manages features, and Workflows or Cloud Composer (managed Airflow) act as orchestration alternatives.

On Azure, Azure ML Pipelines orchestrates, Azure ML Data Labeling annotates, Azure ML Feature Store manages features.

The problem (cross-cutting: prompt + data versioning). Making sure prompt_id, prompt_version and dataset_id, dataset_version propagate through the whole system and show up in spans, runs and metrics. Covered in the posts on prompt versioning and data versioning.

Prompt versioning equivalents. OSS: Langfuse Prompts, MLflow Prompt Registry. AWS: Bedrock Prompt Management (catalogue, versions, labels, integrated A/B testing) and SageMaker Prompt Hub. GCP: Vertex AI Prompt Management inside Vertex AI Studio. Azure: Azure AI Foundry Prompt flow and prompt versioning in Azure OpenAI deployments.

Data versioning equivalents. OSS: DVC + lakeFS (already covered under Data). AWS: S3 Versioning + Lake Formation + Glue Catalog (they are not DVC but together they cover part of it). GCP: Cloud Storage versioning + Dataplex (same). Azure: ADLS Gen2 versioning + Purview (same). The real gap here is that no hyperscaler offers DVC natively, so dataset-as-first-class-citizen operations still require a layer of your own.

Summary table — Retrain stage + cross-cutting pieces.

Functional piece	OSS on-premise	AWS	GCP	Azure
ML pipeline orchestration	Airflow, Dagster, Argo, Kubeflow	SageMaker Pipelines, Step Functions	Vertex AI Pipelines, Cloud Composer	Azure ML Pipelines
Feature store	Feast	SageMaker Feature Store	Vertex AI Feature Store	Azure ML Feature Store
Annotation / HiL	Argilla, Label Studio	SageMaker Ground Truth, A2I	Vertex Data Labeling	Azure ML Data Labeling
Feedback capture	Postgres + Langfuse scores	Bedrock + custom + Ground Truth	Vertex + custom	App Insights + custom
Prompt versioning	Langfuse Prompts, MLflow Prompts	Bedrock Prompt Management	Vertex Prompt Management	Azure AI Foundry Prompt flow
Data versioning	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation	GCS + Dataplex	ADLS + Purview
Cross-system lineage	OpenLineage + DataHub	SageMaker Lineage Tracking	Dataplex lineage	Purview

The chatbot from the previous post ported to AWS

To stop the catalogue being abstract, we take the full scenario from the previous post, the multi-tenant support chatbot for insurers on an on-premise OSS stack, and describe it component by component with an AWS stack. It is not an executable migration; it is the map of what disappears, what appears and where the lock-in shows up.

The network plane. Edge LB and WAF: AWS WAF + CloudFront. Ingress to the cluster: AWS Load Balancer Controller over EKS. What was Cilium BGP + RKE2 is replaced by EKS with VPC CNI (or Cilium on EKS, which is possible). The conceptual equivalent of Tetragon is Amazon GuardDuty for EKS + optional Falco. Moderate lock-in: network control couples to VPC.

The chat gateway and auth. What was a bespoke API gateway with JWT verification materialises as Amazon API Gateway + Amazon Cognito (or IAM Identity Center if it is B2B). The gateway’s AI-aware routing is covered with Bedrock + per-customer tags, or with AWS API Gateway custom authorizers invoking a Lambda for tenant resolution. High lock-in on the identity layer if Cognito is chosen.

The inference engine. Three different options, with a clear trade-off.

Bedrock with a managed model (Claude / Llama / Mistral): all the operations of vLLM, K8s Operators, KV cache and disaggregated serving disappear. You move to Provisioned Throughput for a latency guarantee. You gain time-to-market; you lose control over prefill/decode, over custom LoRA adapters (Bedrock accepts Bedrock-managed fine-tunes but not arbitrary LoRAs), and you enter model lock-in (moving from Claude to Llama means changing API).
SageMaker Endpoints with your own vLLM container: you keep vLLM and its optimisations, but K8s disappears and SageMaker replaces it as the orchestration plane. Inference Components allow you to densify multiple adapters. KV cache, prefix caching and LoRA hot-swap work the same. Moderate lock-in on the SageMaker SDK and on the Inference Components format.
EKS with vLLM (the minimalist option): basically the OSS stack but with EKS instead of RKE2 and EBS/EFS instead of Ceph. Low lock-in, limited benefit from the cloud.

Data layer. The corpus moves to S3 with versioning, the embeddings to Amazon OpenSearch Service or to Aurora pgvector. The radical managed option is Bedrock Knowledge Bases: you upload documents to S3, it indexes them, it exposes a retrieval API. We remove Qdrant, we remove manual embedding pipelines, we remove part of Kafka + Flink. But control over custom reranking, fine-grained per-chunk ACL and the possibility of re-embedding with a new encoder of your own disappears. Bedrock KB uses the Titan or Cohere embedders available in Bedrock, and changing them means changing the whole index. ENS compliance: you have to validate that the buckets and the index live in EU regions and that the embedding model does too.

Stream + CDC. Kafka + Debezium is replaced by MSK + MSK Connect or by Kinesis + DMS. Schema Registry: Glue Schema Registry. The events remain functionally equivalent. Moderate lock-in if you go to Kinesis (Kinesis is not Kafka), low if you go to MSK (Kafka compatibility).

Data versioning. Here the gap is clear. S3 Versioning + Lake Formation + Glue Catalog is not DVC. To preserve the discipline of the previous post, (dataset_id, dataset_version, sha256_hash) propagated as an input artefact to the trainer, you can keep DVC over S3 (DVC works perfectly with S3 as a remote) or accept the limitation and register lineage manually in SageMaker Lineage Tracking. The first option preserves the practice; the second accepts degradation.

Tune stage. The LoRA adapter customer_support_v7 is trained with SageMaker Training Jobs on ml.p5.48xlarge instances (8× H100), using a standard HuggingFace + PEFT container. MLflow managed by SageMaker or your own MLflow on EC2 covers tracking. Alternative: if the black box is acceptable, Bedrock Custom Models with a dataset in S3 produces a fine-tuned Bedrock model without instantiating a GPU manually, at the cost of not being able to inspect the run.

Eval stage. Promptfoo + RAGAS in CI runs the same over CodeBuild. Bedrock Model Evaluation replaces a good part of the automatic eval suite. Bedrock Guardrails replaces NeMo Guardrails + Presidio + LlamaGuard, with the loss of transparency noted earlier.

Deploy stage. If Bedrock was chosen as the engine, this stage vanishes: Bedrock serves. If SageMaker Endpoints + vLLM was chosen, KServe is replaced by SageMaker Operators (or KServe is kept over EKS). The AI Gateway that in OSS was Envoy AI Gateway or LiteLLM becomes API Gateway + Bedrock or API Gateway + Lambda + SageMaker.

Observe stage. The OTel Collector remains the standard. Traces to AWS X-Ray + CloudWatch Application Signals. Metrics to Amazon Managed Prometheus. Logs to CloudWatch Logs + optionally OpenSearch for search. Langfuse is hosted on ECS Fargate or EKS because the cloud has no native equivalent of integrated prompt + traces + scores. Drift: SageMaker Model Monitor replaces Evidently / NannyML. eBPF (Pixie / Hubble / Tetragon) has no direct equivalent in managed AWS, so Falco or installing Tetragon on EKS is still the route.

Retrain stage. SageMaker Pipelines orchestrates the quarterly cycle. SageMaker Ground Truth + A2I replace Argilla. The feedback_signals table in Postgres stays as it is (RDS Postgres) or moves to DynamoDB for large scales.

How much the lock-in weighs. The component with the highest lock-in is Bedrock + Bedrock Knowledge Bases + Bedrock Guardrails: leaving that requires rewriting the inference plane and reindexing the whole RAG. Next is the SageMaker SDK (Pipelines, Endpoints, Training), where leaving costs but means rewriting scripts, not data. Data in S3 is portable (S3 → MinIO with rclone works). OTel observability is portable at almost no cost if the collector is kept as the abstraction. The auth gateway is the other high lock-in point if it uses Cognito.

What you gain. A dramatic reduction in GPU infrastructure operations, K8s patching, CUDA driver management, prefill/decode sizing, Ceph / MinIO management. A very short start-up curve: one request served in less than a sprint versus several weeks of bring-up for the OSS stack. Explicit provider SLAs.

What you lose. Contractual data sovereignty (the data stays in EU regions if configured that way, but the operator is a US third party under the Cloud Act). Visibility of the whole stack (Bedrock is a black box from the model downwards). Roadmap independence (the decision to discontinue a model, raise prices or change guardrails is not controlled by the customer). Fine-grained cost-per-token optimisation (the levers are the ones the provider exposes). For ENS customers under a HIGH declaration or NIS2 essential category, several of these points are non-compliance, not preference.

Master table: the entire parallel catalogue

Stage / component	OSS on-premise (the blog’s reference)	AWS	GCP	Azure
Data	DVC + lakeFS + MinIO + Qdrant + Kafka + Debezium	S3 + Lake Formation + OpenSearch / Aurora pgvector + MSK + DMS	GCS + Dataplex + Vertex Vector Search + Pub/Sub + Datastream	ADLS Gen2 + Purview + Azure AI Search + Event Hubs + ADF
Data versioning (cross-cutting)	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation + Glue Catalog	GCS Versioning + Dataplex lineage	ADLS versioning + Purview
Tune	HF Transformers + PEFT + bitsandbytes + MLflow + Ray/Kubeflow	SageMaker Training + HyperPod + Bedrock Custom + SM Experiments	Vertex AI Training + Vertex Tuning + Vertex Experiments	Azure ML Training + Azure OpenAI fine-tuning + Azure ML + MLflow
Eval	DeepEval + RAGAS + Promptfoo + Langfuse Evals + NeMo Guardrails	Bedrock Model Evaluation + Bedrock Guardrails + SageMaker Clarify	Vertex AI Evaluation Service + Model Armor + Gemini safety	Azure AI Evaluation SDK + Content Safety (Prompt Shields, Groundedness)
Deploy	vLLM + KServe + LLM Operators + Envoy AI Gateway	Bedrock + SageMaker Endpoints (+ Inferentia / Trainium)	Vertex AI Prediction + Gemini API (+ TPU)	Azure OpenAI + Azure ML Endpoints (+ Maia)
Observe	OTel + Tempo + Prometheus + Loki + Langfuse + Phoenix + Hubble	CloudWatch + X-Ray + ADOT + AMP/AMG + SM Model Monitor	Cloud Monitoring + Cloud Trace + Vertex Model Monitoring	Azure Monitor + App Insights + Azure ML Model Monitor
Retrain	Airflow / Argo / Kubeflow Pipelines + Argilla + Feast	SageMaker Pipelines + Ground Truth + A2I + SM Feature Store	Vertex AI Pipelines + Data Labeling + Vertex Feature Store	Azure ML Pipelines + Data Labeling + Azure ML Feature Store
Prompt versioning (cross-cutting)	Langfuse Prompts + MLflow Prompt Registry	Bedrock Prompt Management + SM Prompt Hub	Vertex AI Prompt Management	Azure AI Foundry Prompt flow

When to choose each side — the real decision

The right question is not “OSS or cloud?”. It is stage by stage.

The OSS side wins by default when there is:

Data subject to ENS HIGH category, NIS2 essential sectors or equivalents (identifiable health data, regulated banking, critical infrastructure). Here provider traceability and the processing contract are not negotiable; using a service whose operator is subject to the Cloud Act, FISA 702 or equivalent compromises the legal basis.
A requirement for auditable inspection of the model, the guardrails and the whole pipeline. If a regulator asks “how exactly does it detect PII?” and an answer that ends in open code is mandatory.
Large volumes with stable workloads. Above a certain threshold of tokens per month, the cost of Bedrock / AOAI / Vertex diverges from the amortised cost of your own GPU cluster. The threshold depends on the workload, but it typically sits between 5 and 50 billion tokens per month for models in the Llama 70B range.
Roadmap independence as a priority. The day the provider discontinues a model or raises the price by 40 %, the organisation has to be able to ignore it.

The hyperscaler side wins by default when there is:

Critical time-to-market, an MVP in weeks. The operational weight of the OSS stack is too much for a project that has not yet proved product-market fit.
A small team with no SREs / MLEs specialised in GPU inference. Operating KServe + vLLM + KV cache + multi-tenancy is not trivial; if the team cannot sustain it, hosting is the way.
Variable workloads / unpredictable spikes. Bedrock on-demand and SageMaker serverless charge for what you use; your own cluster pays for the GPU whether it is busy or not.
A need for specific proprietary models (Claude, GPT-4.1, Gemini Pro) with no acceptable OSS equivalent for the case.

Mixed stages are frequent and reasonable. In practice, a common pattern in 2026 is: data, observe and retrain in self-hosted OSS (lineage and sovereignty), tune in OSS on your own cluster, eval in OSS + managed guardrails depending on the safety profile, deploy managed for proprietary models and self-hosted for open models. The question to ask for each stage is: “if the provider raises prices by 50 % or discontinues a component tomorrow, how much does it cost to move it?”. The parallel catalogue in this post gives the answer for each box.

What we have not covered (yet)

Some pieces still deserve a post of their own:

OpenAI / Anthropic API directly (not through Bedrock or AOAI): another level of managed, another contract.
Serious hybrids: AWS Outposts, GCP Distributed Cloud, Azure Stack HCI / Azure Local, the hyperscaler in your own room.
Per-tenant cost accounting compared OSS vs cloud: how the bill is built and where attribution breaks.
A real migration OSS → cloud or cloud → OSS: steps, scripts, gotchas.
Concrete European sovereignty: GAIA-X, EuroHPC, the European cloud offering (OVHcloud, Scaleway, IONOS, Aruba), compared with the big three for ENS / NIS2 cases.
AWS Inferentia / Trainium, GCP TPU v6 Trillium, Azure Maia: in-house chips and how they change the cost / token calculation.

References

vLLM documentation and vLLM Production Stack — the reference OSS inference engine.
Amazon Bedrock documentation — AWS managed model catalogue, Knowledge Bases, Guardrails and Prompt Management.
Amazon SageMaker AI — training, endpoints, pipelines, model monitoring.
Google Vertex AI documentation — training, prediction, evaluation, model monitoring.
Azure AI Foundry documentation — Microsoft’s unified plane for AI applications.
Azure OpenAI Service documentation — OpenAI models hosted on Azure.
OpenTelemetry GenAI semantic conventions — the standard that stitches observability across the OSS / cloud boundary.
Langfuse documentation and Arize Phoenix — the reference OSS LLM observability tools.
DVC and lakeFS — OSS data versioning.
NeMo Guardrails — OSS safety + dialog policy.
ENS (Esquema Nacional de Seguridad) and NIS2 (Network and Information Security Directive 2) — the compliance frameworks that have the last word in the OSS vs cloud choice for regulated EU customers.

Sovereignty on lo0 — Tech Blog

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

TL;DR

The analogy: the order form against the shopping list

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

The OAM model in five pieces

CUE, the power and the toll

The vela CLI: Heroku on the outside, GitOps on the inside

Multi-cluster, which is what matters to you

VelaUX, the lightweight portal thrown in

Score: describe the workload once

What it is and what it solves

The two implementations

Score against KubeVela and Helm

Where each one fits: the cross-cutting comparison

The fit with inference: a pattern, not an official recipe

Caveats, no frills

For your sovereign inference factory

See also

Sources

Crossplane: the control plane that turns your cluster into a private cloud API

TL;DR

The analogy: the thermostat against the light switch

What Crossplane is and what state it reaches 2026 in

The v2 turn (which changes how you think about it)

The model, piece by piece

Composition functions: the real logic

Crossplane against Terraform, OpenTofu and Pulumi

The awkward question: is it an AWS-style CLI?

On-premise providers: provisioning without touching AWS

The fit with GPU inference: from pattern to Modelplane

What it costs you to operate your own control plane

How it lives alongside what you already have

For your sovereign inference factory

See also

Sources

From public cloud to private: what you really lose and which projects fill the gap

TL;DR

The analogy: leaving the hotel to set up your own house

Why this conversation belongs to 2026, not to always

What you lose when you leave the cloud, specifically

The map of projects that fill each gap

The gap you will not close

When NOT to repatriate

The numbers, to ground it

For your sovereign inference factory

See also

Sources

Sovereign on-premise vs hyperscalers: the case with data (cost, energy, performance and sovereignty)

What this article covers

The framework: four axes, not one number

The cost axis: TCO and break-even

Three-year TCO: the full calculation for an 8×H100 node

The uncomfortable reality: the utilisation almost nobody reaches

The hidden costs of the cloud: egress

The performance axis: the provider does not decide, goodput does

The energy axis: the European and Spanish advantage

The sovereignty axis: the one that does not depend on utilisation

The scorecard: the three options scored

When each option wins

Sizing the hybrid: how much on iron, how much on cloud

Assumptions and sensitivity

Decision checklist

Limits and traps (data-driven)

See also

Sources

From GPU-hour to cost per token: the metric that compares on-prem and cloud

What this article covers

The identity: from GPU-hour to token

The two halves of cost per token

The gateway: LiteLLM in depth

The self-hosted trap: LiteLLM does not know what your own token costs you

Configuring custom pricing: an example

Cost per token changes with throughput: read it off the sweep

Cost per request: not only per token

Cost per model and per tier

Attribution: from cost to team and product

A combined report example

Governing spend: budgets, caps and alerts