Finops on lo0 — Tech Blog

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

From public cloud to private: what you really lose and which projects fill the gap

Mon, 31 Aug 2026 09:15:00 +0200

With this one I open a run about the biggest gap a sovereign platform leaves you: how to give your teams the self-service experience of a public cloud, without the public cloud. Here you have the map of the problem; the next two go down into the detail of the tools, Crossplane as the infrastructure control plane and KubeVela with Score as application-centric self-service.

TL;DR

Leaving the public cloud has a solid business case, and I am not making it up. 37signals declared savings of close to two million dollars in 2024 after repatriating, the classic a16z analysis puts cloud spend at around half of the cost of revenue of a software company, and the Barclays CIO survey of 2024 recorded the highest repatriation intent in its history. If your workload is a predictable, sustained base (which is exactly the continuous LLM inference this blog is about) the numbers work out, and in Europe the sovereignty reasons pile on top: the Data Act, the tension with the US CLOUD Act, and that dependence on three hyperscalers that control around 70 % of the European cloud market.

But the problem is not the compute. Your on-premise GPU rack serves tokens just as well as a rented one. What the cloud was selling you and you now have to rebuild is something else: the elasticity of asking for a hundred machines at ten and handing them back at eleven, the catalogue of hundreds of managed services with an on-call team behind them, the unified provisioning API with fine-grained permissions, the multi-region network, the bill that arrives itemised, and that model where operating all of it was somebody else’s problem. All of that you rebuild piece by piece with open source projects, and the rebuild has a cost that the savings figures do not usually tell you about. What follows is the map of what you lose, which project fills each gap, and where a gap remains that nothing will close completely.

The analogy: leaving the hotel to set up your own house

Think of it like this: living in the public cloud is living in a hotel. You pay a lot per night, but you do not change light bulbs, reception is open at three in the morning, if you need another room you ask for it and it appears, and when you leave you take nothing with you to maintain. Setting up your own house works out much cheaper per square metre if you are going to stay for years, and on top of that you decide who comes in and where your things are kept. The problem arrives the day a pipe bursts: then you do not call reception, because reception is you.

Repatriation is that move. The savings are real and so is data ownership, but what in the hotel came included in the price of the night is now your to-do list: the plumbing is your network, the boiler is your storage, the concierge who got you anything is the catalogue of managed services you no longer have, and the itemised bill they slid under your door is a cost model you have to build, because electricity and depreciation do not arrive broken down per tenant. The house pays off, but only if you count what it costs to furnish it and fix the pipes, not just the price per square metre against the hotel.

The rest of the article walks through that to-do list and tells you, for each item, which open source tool does the job the hotel used to do for you.

Why this conversation belongs to 2026, not to always

Repatriating is nothing new, but in 2026 three things come together that make it urgent if you run a European AI platform.

The first is economic, and Andreessen Horowitz put it on the table in 2021 with The Cost of Cloud, a Trillion Dollar Paradox. Their thesis, which you should read knowing that a16z has interests in the ecosystem it analyses, is that cloud spend averages around 50 % of the cost of revenue in a software company, and that recovering half of that spend would free up margins the market penalises. The cleanest case, and one you can verify, is 37signals: DHH published that their cloud bill dropped from 3.2 to 1.3 million dollars a year after repatriating, with an initial investment of about 700,000 dollars in Dell servers. Now the honest caveat, which the specialist press itself underlines: those savings figures did not include the extra operations staff, nor the energy, nor the cooling. I come back to it in the hidden costs section.

The second is declared intent. The Barclays CIO survey from the first half of 2024 recorded that 83 % planned to repatriate some workload to private cloud or on-premise in the following twelve months, the highest reading in the series. But read that 83 % precisely: it is the proportion of managers who plan to move some workload, not the percentage of workloads or of spend that moves. Flexera, in its 2025 report, gives you the more grounded figure: large companies have repatriated around 21 % of the workloads they had in the public cloud, and cost saving is priority number one for almost 60 %.

The third is regulatory and European, and it is the one that turns repatriation into something more than a FinOps decision. The European Union Data Act came into force in January 2024 and has applied since September 2025; it mandates portability between providers and, from January 2027, bans the exit fees that today chain you to your provider. And underneath it beats a deeper legal tension: the US CLOUD Act allows United States authorities to demand data from an American provider even if it is stored in Frankfurt, which is exactly the argument for why a European region of a hyperscaler does not give you sovereignty. With AWS, Microsoft and Google controlling around 70 % of the European cloud market, the conversation about sovereign cloud stopped being theoretical. I developed this already in sovereign on-premise versus hyperscalers.

What you lose when you leave the cloud, specifically

Let us get to the core, ordered by what is going to hurt you most. And one thing before starting: none of these gaps is about compute. They are all about what the cloud put around the compute.

Hardware elasticity disappears. On-premise, the GPU you did not buy does not exist. The cloud bills you by the hour precisely because it takes on the risk of idle capacity: it lets you ask for a peak and hand it back. With your own iron, you buy the peak up front and pay for it whether it is in use or not, so your profitability depends on high sustained utilisation. Below a certain usage threshold, the cloud beats you again. This is the irreducible gap, the one no tool fills, and if you repatriate a workload with extreme peaks you are making the wrong decision.

Managed services become your operations. Every box you took for granted (the database, the queue, the object storage, the load balancer, DNS, the secrets manager) stops being a checkbox you tick and becomes a system you install, update and watch yourself. The 37signals case teaches it well: getting out of S3 object storage was the last and hardest part, with petabytes of data and a contract they dragged along for years. Your managed Postgres turns into operating CloudNativePG; your S3 turns into operating MinIO; and so on with every service.

You have to build the provisioning control plane yourself. AWS gives you a unified API where you ask for a resource with fine-grained permissions and it appears. On-premise that does not come for free: you have to build the control plane that translates “I want a database” into real resources. It is exactly the gap that Crossplane fills, and that is why it has its own article.

You replace the managed network piece by piece. The VPC, the security groups, NAT, the load balancer: every network abstraction in the cloud has its on-premise equivalent (Cilium for network policies on eBPF, MetalLB for the load balancer without ELB, Gateway API for L7 ingress), but they are pieces you assemble, not a service you switch on. A good part of this we already saw in the networking vertical with Cilium and eBPF.

Billing stops existing as such. In the cloud the cost reaches you broken down by service and by tag; you have a Cost Explorer that answers “how much does this team cost me”. On-premise there is no natural bill: there is hardware depreciation, an electricity bill and some payroll, and splitting that per workload is a model you have to build. OpenCost helps you with the split per workload inside the cluster, but the total physical cost (depreciation plus energy plus staff) you model separately, as we saw in OpenCost and cost allocation.

You design multi-zone availability yourself. The three availability zones the cloud gives you for free turn into several sites, several racks and several utility feeds that you have to plan. 37signals solved it with two datacenters and replication between them. It is neither free nor automatic.

And above all, the on-call is yours. The cloud was not selling you only compute: it was selling you operations included. When something falls over at three in the morning, in the cloud there is an on-call team at the provider; with open source, that team is you. This is the cost that savings spreadsheets almost never include, and it is the one that decides whether your repatriation was a good idea or a hidden bill.

The map of projects that fill each gap

With the gaps ordered, the catalogue of substitutes reads itself. I group it by layer, from the lowest to the closest to the developer.

In the infrastructure substrate, what replicates EC2 virtual machines, virtual networking and disks, you have OpenStack as the classic standard for a complete private cloud, SUSE’s Harvester as bare-metal hyperconvergence on KubeVirt managed from Kubernetes, and emerging projects such as Spinifex that rebuild an AWS-compatible API on your own iron. It is the heaviest layer and the one I touch least in this blog, because I assume your cluster already exists.

In the provisioning control plane, what replicates the AWS IAM and resource API, there is Crossplane, graduated in the CNCF in November 2025, which brings the Kubernetes declarative API model to infrastructure. Alongside it, Cluster API for declarative cluster provisioning. And so you can see how far this reaches in our own field, the Crossplane team itself published Modelplane in mid-2026, an inference control plane built on compositions that serves models with vLLM. This layer is developed by the Crossplane article.

In the developer self-service experience, what replicates the AWS console and CLI, you have three different approaches you should not confuse. Backstage is the web portal, the catalogue and the templates, and it already has its own article. KubeVela brings self-service to the application model with a friendly CLI. Score defines a workload specification that is portable between environments. KubeVela and Score share the third article because they cover the same layer from angles that complement each other.

In on-premise managed services, CloudNativePG operates Postgres for you with high availability and backups to object storage, MinIO gives you S3-compatible storage, and OpenCost covers that cost allocation you lost with Cost Explorer.

The way to read this map is that none of these pieces is surplus if you want the cloud experience: they are the boxes the cloud ticked for you and that you now tick one by one.

The gap you will not close

I will tell you plainly, because it is the part the “leave the cloud” brochures skip. No combination of open source projects reproduces 100 % of the public cloud for you, and not for lack of maturity, but by design.

What you will not replicate is, first, instant hardware elasticity: you still buy the peaks up front. Second, the depth and breadth of the managed services catalogue, hundreds of services with a service level agreement and an on-call team behind them, which no internal team matches. Third, the global multi-region network the cloud gives you almost for free. And fourth, that model of transferred operational responsibility: with open source, operations come back home.

The correct reading is not that repatriating is a bad idea, but that you swap a predictable operating expense and a delegated responsibility for a cheaper capital investment and a responsibility of your own. For sustained base load it works out. For everything else, think about it.

When NOT to repatriate

This section has to be here, because repatriation badly applied is expensive. There are three clear signs that the cloud is still your right answer.

The first is a workload with extreme peaks or very variable traffic. Most inference deployments operate between 40 and 65 % GPU utilisation, and below roughly 70 % sustained usage the cloud beats you on total cost. If your workload has long troughs, you are buying iron so it can sleep.

The second is uncertainty or the start of a project. The break-even point in favour of buying only applies to you when utilisation is consistently high and, above all, measured and not projected. Buying GPUs against an optimistic forecast is the fastest way to end up with a warehouse of idle silicon. Sizing it properly is worth a whole article: sizing and justifying the GPU investment.

The third is the lack of an operations team. The total cost of ownership on-premise includes between half and one full-time engineer per GPU server. Without that team, your saving is illusory, because the on-call the cloud included is now covered by nobody. a16z itself frames its thesis as a hybrid approach, not as “leave the cloud” at all.

The numbers, to ground it

So as not to leave you with the discussion in the abstract, here are the orders of magnitude, triangulating two sources with opposite biases, a cloud provider and a neutral FinOps tool. An H100 in the cloud runs you between 2.90 and almost 7 dollars per GPU-hour depending on provider; bought, an H100 card is around 31,000 dollars and an eight-way system between 250,000 and 320,000. At full utilisation, a rented H100 passes its purchase price in less than a year, and an eight-way system with second-hand hardware can pay for itself in about seven months. The three-year total cost of an eight-H100 system, counting staff, colocation, energy and cooling, is estimated at between 700,000 and 950,000 dollars, and the threshold where on-premise beats the hyperscalers is around 80 % sustained utilisation.

The two sources disagree on the exact number, as you would expect, but they agree on the deciding factor: sustained utilisation, measured and not projected. The full working of this calculation is in the TCO of an on-premise GPU cluster. Here the conclusion is enough for you: for continuous base load the numbers work out, and that is exactly the profile of an inference factory.

For your sovereign inference factory

Everything above arranges itself into a practical decision. If you operate an LLM inference platform on your own iron, you are not deciding whether to repatriate (you already took that decision when you built the cluster), but how to rebuild the cloud experience your teams miss.

The sensible sequence starts with the substrate, which you probably already have in the form of RKE2 and storage. It goes on to the provisioning control plane, where Crossplane turns “open a ticket and wait” into a declarative API. It continues with the developer experience, where you choose between the Backstage portal, the KubeVela application model or the portable Score specification, depending on whether your people prefer a web or a CLI. And do not forget the on-premise managed services or the cost allocation, because without a bill your platform loses the discipline the cloud imposed by default.

The next two articles go down into the two layers least covered in this blog: Crossplane for the infrastructure control plane, and KubeVela with Score for CLI self-service centred on the application. With them, the map of this layer is covered, and your own house starts to look like the hotel you left, with the difference that the data stays inside.

Sources

Andreessen Horowitz (S. Wang, M. Casado), The Cost of Cloud, a Trillion Dollar Paradox — https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/
Data Center Dynamics, 37signals claims it saved almost 2M USD last year from cloud repatriation — https://www.datacenterdynamics.com/en/news/37signals-claims-it-saved-almost-2m-last-year-from-cloud-repatriation/
The Register, Developer pockets 2M USD in savings from going cloud-free — https://www.theregister.com/2024/10/21/37signals_aws_savings/
Barclays, Technology: 1H24 CIO Survey (PDF) — https://8198920.fs1.hubspotusercontent-na1.net/hubfs/8198920/Barclays_Cio_Survey_2024-1.pdf
Channelnomics, Breaking Down the 83% Public Cloud Repatriation Number — https://channelnomics.com/breaking-down-the-83-public-cloud-repatriation-number/
The New Stack, Updated Stats on Cloud Sustainability, Repatriation and Cost Optimization (Flexera 2025) — https://thenewstack.io/updated-stats-on-cloud-sustainability-repatriation-and-cost-optimization/
Comisión Europea, Data Act | Shaping Europe’s digital future — https://digital-strategy.ec.europa.eu/en/policies/data-act
EU Data Act, Article 29 — Gradual withdrawal of switching charges — https://www.eu-data-act.com/Data_Act_Article_29.html
Kiteworks, How the EU Data Act and GDPR Conflict with U.S. CLOUD Act — https://www.kiteworks.com/gdpr-compliance/eu-data-act-gdpr-cloud-conflict/
Computerworld, EU takes first steps to reduce reliance on US hyperscalers — https://www.computerworld.com/article/4181816/eu-takes-first-steps-to-reduce-reliance-on-us-hyperscalers.html
CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Harvester HCI (SUSE) — https://harvesterhci.io/
CloudNativePG — https://cloudnative-pg.io/
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
Spheron, LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
CloudZero, H100 GPU Cost in 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

Tue, 16 Jun 2026 11:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator (1{,}234). Generic example hardware: 4×H100 SXM5 80 GB nodes. The dollar sign is not used (it is the formula delimiter).

TL;DR

With an SLO of TTFT P99 ≤ 300 ms and ITL P99 ≤ 50 ms for a production chat, peak GPU utilisation must stay at ≤ 63 %.
A service receiving 5M requests/day with 512 input tokens and 256 output tokens on average generates an hourly peak of ~11,600 tok/s (with a peak factor of 1.8×). With vLLM on H100 SXM5 (Llama-3.3 70B FP8, ~1,850 tok/s at 50 concurrent requests), ≥ 10 GPUs are needed at peak to meet the SLO, which translates into 3 4×H100 nodes (with headroom).
The all-in cost of those 3 nodes is ~83,300 € per year per node (3-year capex + base opex); the €/1M tokens in the base scenario at 63 % utilisation is ~0.37 €.
Against AWS p5 on-demand (~3.90 USD/GPU-hour after the 44 % cut of Jun. 2025), the utilisation break-even sits at ~55 %; against neocloud on-demand (~2.01 USD/GPU-hour), there is no feasible break-even at medium utilisation.
Capex payback happens between 13 and 24 months if sustained utilisation exceeds 70 %.

1. The sizing chain: from demand to the number of GPUs

Sizing an inference platform follows a causal chain of five steps. Each step has a formula; none can be skipped.

1.1 Step 1 — Characterise the demand

Parameter	Symbol	Example
Requests per day (average)	$D$	5,000,000
Input tokens per request (average)	$L_{\text{in}}$	512
Output tokens per request (average)	$L_{\text{out}}$	256
Peak factor (peak hour vs average ratio)	$k_{\text{peak}}$	1.8
Hours per day with significant traffic	$H$	16

The average output throughput:

$$\dot{T}_{\text{avg}} = \frac{D \times L_{\text{out}}}{86{,}400\,\text{s}} = \frac{5{,}000{,}000 \times 256}{86{,}400} \approx 14{,}815\;\text{tok/s}$$

The peak throughput (busy hour):

$$\dot{T}_{\text{peak}} = k_{\text{peak}} \times \frac{D \times L_{\text{out}}}{H \times 3{,}600} = 1.8 \times \frac{5{,}000{,}000 \times 256}{16 \times 3{,}600} \approx 40{,}000\;\text{tok/s}$$

Note: if the traffic profile has very pronounced peaks (peak/average ratio > 3), sizing is done for the peak and average utilisation falls; the sensitivity analysis in §5 quantifies the effect.

1.2 Step 2 — Set the SLO and derive the maximum utilisation

The latency SLO imposes a ceiling on GPU utilisation. Using queueing theory (M/G/1 model):

$$\rho_{\max} \approx 1 - \frac{1}{\sqrt{1 + C_{s}^{2}}} \cdot \frac{W_{\text{queue}}^{*}}{\bar{s}}$$

where $\rho$ is the utilisation, $W_{\text{queue}}^{*}$ the maximum admissible queueing time and $\bar{s}$ the average service time per request. For the simplified M/M/1 case with prefill time dominant:

$$\rho_{\max} = 1 - \frac{W_{\text{queue}}^{*}}{\bar{s} \cdot (1 + W_{\text{queue}}^{*}/\bar{s})}$$

The practical reference table (derived from the Spheron/Little’s Law formula, 2026):

SLO TTFT P99	Maximum utilisation $\rho_{\max}$
200 ms	55 %
300 ms	63 %
400 ms	70 %
500 ms	75 %

For our example (SLO 300 ms P99): $\rho_{\max} = 0.63$.

The ITL SLO imposes an additional constraint: the inference engine must be able to generate the next token in ≤ 50 ms. On H100 SXM5 with vLLM and Llama-3.3 70B FP8, the ITL P50 at 50 concurrent requests is ~20 ms, with P99 ~45 ms. ITL is the dominant constraint only when the batch size is very high (>64 sequences) or VRAM is nearly full.

1.3 Step 3 — Required throughput and throughput per GPU

The throughput the cluster must serve at peak, while respecting $\rho_{\max}$:

$$\dot{T}_{\text{required}} = \frac{\dot{T}_{\text{peak}}}{\rho_{\max}} = \frac{40{,}000}{0.63} \approx 63{,}500\;\text{tok/s (installed capacity)}$$

The throughput per GPU (reference benchmark, vLLM v0.18.0, Llama-3.3 70B FP8, H100 SXM5 80 GB, 50 concurrent requests):

Engine	Throughput (tok/s per GPU)	TTFT P50 / P95 (50 req)	Source
vLLM 0.18.0	1,850	380 ms / 720 ms	Spheron benchmarks, Mar. 2026
SGLang 0.5.9	1,920	360 ms / 680 ms	Spheron benchmarks, Mar. 2026
TensorRT-LLM 1.2.0	2,100	340 ms / 620 ms	Spheron benchmarks, Mar. 2026

The throughput data correspond to the test with Llama 3.3 70B Instruct FP8, 512 input / 256 output tokens, 50 concurrent requests, on bare-metal H100 SXM5. See the engine analysis in serving engines Pareto comparison.

We use vLLM as the general-purpose production reference: $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s.

1.4 Step 4 — Number of GPUs and number of nodes

$$N_{\text{GPU}} = \left\lceil \frac{\dot{T}_{\text{required}}}{\dot{T}_{\text{GPU}}} \right\rceil = \left\lceil \frac{63{,}500}{1{,}850} \right\rceil = \lceil 34.3 \rceil = 35\;\text{GPUs}$$

With 4×H100 SXM5 nodes:

$$N_{\text{nodes}} = \left\lceil \frac{N_{\text{GPU}}}{4} \right\rceil = \left\lceil \frac{35}{4} \right\rceil = 9\;\text{nodes}$$

We add 15 % headroom for hardware failures (~5 % annual rate in small clusters), upgrades and unforeseen peaks:

$$N_{\text{nodes, final}} = \lceil 9 \times 1.15 \rceil = 11\;\text{nodes} \approx 44\;\text{GPUs}$$

For the TL;DR example (5M requests/day with a 16 h active profile, k 1.8 and an SLO of 300 ms) the number of nodes is 11. The simplified TL;DR case with k=1 and H=24 gives 3 nodes; the difference illustrates the impact of the hourly profile.

1.5 Summary of the chain

2. From sizing to cost: the TCO model

With $N_{\text{nodes}} = 11$ 4×H100 SXM5 nodes, the TCO follows the model detailed in TCO of an on-premise GPU cluster. Here the compact formula is reproduced and applied to the sized cluster.

2.1 Annual cost per node (base scenario)

$$C_{\text{node/year}} = \frac{\text{node capex}}{\text{years}} + \text{node opex/year}$$

Line item	Value (USD / €)	Note
4×H100 node capex (midpoint)	178,500 USD	GPUs + server + network + storage + rack
3-year amortisation	59,500 USD/year ≈ 55,300 €/year	Straight line
Opex/year (base scenario, 8–16 node cluster)	~28,000 €/year	Energy + staff + maintenance + colocation
Total per node per year	~83,300 €/year	—

For 11 nodes: 915,300 €/year of total fixed cost.

2.2 From €/node-year to €/GPU-hour

$$\text{EUR/GPU-hour} = \frac{C_{\text{node/year}}}{4\;\text{GPUs} \times 8{,}760\;\text{h} \times \rho}$$

Utilisation $\rho$	EUR/GPU-hour
40 %	5.93
55 %	4.31
63 % (SLO 300 ms)	3.76
70 %	3.39
80 %	2.97
100 %	2.38

2.3 From €/GPU-hour to €/1M tokens

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^{6}}{\dot{T}_{\text{GPU}} \times 3{,}600}$$

With $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s (vLLM, Llama-3.3 70B FP8):

Utilisation	EUR/GPU-hour	EUR/1M tokens
40 %	5.93	0.891
55 %	4.31	0.647
63 % (SLO 300 ms)	3.76	0.564
70 %	3.39	0.509
80 %	2.97	0.446
100 %	2.38	0.357

The complete cost/token identity as a function of throughput and utilisation is developed in cost per token and per request.

2.4 Comparison with cloud (equivalent €/1M tokens)

To compare, the cloud price is converted to the €/1M tokens equivalent using the same reference throughput ($\dot{T}_{\text{GPU}} = 1{,}850$ tok/s):

$$\text{EUR/1M tokens (cloud)} = \frac{P_{\text{cloud}} \times 10^{6}}{1{,}850 \times 3{,}600}$$

Provider / Mode	GPU-hour price (USD)	EUR/GPU-hour ($1\,\text{USD} \approx 0.93\,\text{EUR}$)	EUR/1M tokens equiv.
AWS p5 on-demand (post Jun. 2025)	3.90	3.63	0.545
AWS p5 1-year reserved	~2.50	2.33	0.350
CoreWeave on-demand	~2.01	1.87	0.281
CoreWeave 3-year reserved	~1.49	1.39	0.209
GCP A3 on-demand	~3.67	3.41	0.512
Azure ND H100 v5 on-demand	~6.98	6.49	0.975

Sources: IntuitionLabs (Jun. 2026), CloudZero (Jun. 2026), Spheron GPU pricing (May 2026).

3. Break-even and payback of your own investment

3.1 The utilisation break-even formula

The utilisation break-even $u^{*}$ is the utilisation at which the annual on-prem cost per GPU-hour equals the cloud price:

$$u^{*} = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

where $P_{\text{cloud}}$ is the cloud price in the same currency as the on-prem costs.

With the base scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/node/year):

Cloud reference	Cloud price (EUR/GPU-hour)	$u^{*}$ break-even
Azure on-demand (~6.49 EUR)	6.49	23 %
AWS on-demand post-cut (~3.63 EUR)	3.63	41 %
GCP on-demand (~3.41 EUR)	3.41	44 %
CoreWeave on-demand (~1.87 EUR)	1.87	80 %
CoreWeave 3-yr reserved (~1.39 EUR)	1.39	>100 % (impossible)
AWS 1-yr reserved (~2.33 EUR)	2.33	64 %

3.2 Capex payback

Payback is the time $T_{\text{pay}}$ in which the cumulative saving against cloud equals the initial capex:

$$T_{\text{pay}} = \frac{\text{total cluster capex}}{(\text{cloud cost/year}) - (\text{on-prem opex/year})}$$

where the cloud cost/year is computed at the same sustained utilisation.

For the 11-node cluster (total capex 11 × 178,500 USD ≈ 1,825,000 USD ≈ 1,697,000 €):

Cloud reference	Sustained utilisation	Annual saving vs cloud	Payback
AWS on-demand (3.63 EUR/h)	70 %	(3.63 − 3.39 EUR) × 4 × 8760 × 0.70 × 11 nodes ≈ 63,900 €/year	~27 months
AWS on-demand (3.63 EUR/h)	80 %	(3.63 − 2.97) × 4 × 8760 × 0.80 × 11 ≈ 203,600 €/year	~10 months
GCP on-demand (3.41 EUR/h)	70 %	(3.41 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 5,400 €/year	~315 months (not viable)
Azure on-demand (6.49 EUR/h)	70 %	(6.49 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 830,000 €/year	~2 months

The 13-month payback quoted in studies such as Lenovo TCO 2026 corresponds to ~80 % utilisation against high-priced hyperscalers (Azure/AWS before the cut). With current prices (post June 2025, AWS at 3.90 USD), the window widens.

3.3 Simple payback (capex only vs equivalent cloud)

If on-prem opex is omitted and only capex is compared with the gross saving:

$$T_{\text{pay,simple}} = \frac{\text{capex}}{P_{\text{cloud}} \times 4 \times 8{,}760 \times \rho \times N_{\text{nodes}}}$$

Utilisation	AWS (3.63 EUR)	Azure (6.49 EUR)
50 %	54 months	30 months
70 %	39 months	22 months
80 %	34 months	19 months

4. Decision table: when to buy, rent or go hybrid

The table below is a five-dimension Pareto. There is no implicit order between columns; the reading depends on the organisation’s constraints.

Option	EUR/GPU-hour (util. 70 %)	Initial capex	Required utilisation	Data sovereignty	Peak elasticity
On-prem purchase, util. ≥ 70 %	3.39	high (178k USD/node)	≥ 70 % sustained	total	none
On-prem purchase, util. < 50 %	> 4.75	high	< 50 % → loses vs cloud	total	none
Sovereign EU cloud OD (Scaleway/Nebius EU)	2.00–3.59	none	any	yes (EU)	total
AWS p5 on-demand (post Jun. 2025)	3.63	none	any	no (CLOUD Act)	total
CoreWeave on-demand	1.87	none	any	partial (US)	total
CoreWeave 3-year reserved	1.29–1.39	financial commitment	rigid contract	partial (US)	none
AWS 1-year reserved	~2.17	1-year commitment	contract	no (CLOUD Act)	none
Hybrid on-prem base + EU cloud peak	2.50–3.39 (weighted)	medium	base ≥ 70 %, elastic peak	yes (EU)	elastic peak

Cut-off criteria that come before the table:

GDPR sovereignty: if the data is personal or the system is high-risk under the EU AI Act, CoreWeave/AWS are ruled out before any price comparison.
Minimum volume to amortise capex: below ~2M tokens/day sustained over 3 years, on-prem capex does not amortise against AWS on-demand.
Traffic elasticity: peaks >3× the base favour a hybrid or pure cloud; a stable base favours on-prem.

The cost/sovereignty Pareto frontier for GDPR data leaves three options: on-prem, sovereign EU cloud and hybrid. Sustained utilisation and traffic predictability decide between them. See the cross-analysis of the four axes in sovereign on-premise vs hyperscalers.

5. Sensitivity analysis

5.1 Sizing vs hourly profile and peak factor

The number of GPUs grows linearly with $k_{\text{peak}}$ and inversely with $\rho_{\max}$:

$$N_{\text{GPU}} = \left\lceil \frac{D \times L_{\text{out}} \times k_{\text{peak}}}{H \times 3{,}600 \times \rho_{\max} \times \dot{T}_{\text{GPU}}} \right\rceil$$

Peak factor $k$	SLO 300 ms ($\rho_{\max}=0.63$)	SLO 500 ms ($\rho_{\max}=0.75$)
1.2	8 GPUs (2 nodes)	6 GPUs (2 nodes)
1.8	12 GPUs (3 nodes)	10 GPUs (3 nodes)
2.5	16 GPUs (4 nodes)	13 GPUs (4 nodes)
3.5	22 GPUs (6 nodes)	18 GPUs (5 nodes)

(Example simplified to 5M req/day with H=24 to illustrate the sensitivity to the peak factor)

A peak factor of 3.5× triples the number of nodes compared with k=1.2 at the same SLO. Sizing hardware for $k > 2.5$ leaves GPUs idle more than 70 % of the time; peak cloud is more efficient beyond that threshold.

5.2 Break-even vs sustained utilisation

$$u^{*} = \frac{83{,}300}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

Cloud price (EUR/GPU-hour)	$u^{*}$ break-even	Scenario
6.49 (Azure OD)	23 %	On-prem wins almost always
3.63 (AWS OD)	41 %	On-prem wins if util. > 41 %
3.41 (GCP OD)	44 %	—
2.33 (AWS 1-yr reserved)	64 %	On-prem wins if util. > 64 %
1.87 (CoreWeave OD)	80 %	Hard to reach in production
1.39 (CoreWeave 3-yr reserved)	>100 %	On-prem never closes the gap

5.3 Break-even vs energy price

Energy accounts for 6–11 % of total TCO. Its impact on the break-even is moderate:

Energy price (EUR/kWh)	Energy opex/year per node	EUR/GPU-hour (70 % util.)	$u^{*}$ vs AWS OD
0.034 (solar PPA)	1,604 €	3.22	38 %
0.116 (industrial ES, base)	5,475 €	3.39	41 %
0.200 (high tariff)	9,437 €	3.57	43 %

The difference between the cheapest and the most expensive scenario is only 5 percentage points of break-even. The variable that moves the needle is utilisation, not energy.

5.4 Break-even vs demand growth

If demand grows at an annual rate $g$, the average utilisation of the cluster (sized for year 1) rises over time until it saturates and has to be expanded:

$$\rho(t) = \rho_{0} \times (1 + g)^{t}$$

Annual growth $g$	Time to saturation ($\rho \to 100\,\%$)	Decision
10 %	~11 years	Comfortable purchase
30 %	~4 years	Purchase with a review at 3 years
60 %	~2 years	Hybrid: base + elastic cloud
>100 %	<1 year	Pure cloud until it stabilises

For growth above 30 % per year, a buy-only strategy means oversizing for the future peak or re-buying hardware in short cycles. The hybrid (on-prem base + cloud for growth) minimises the capex at risk.

5.5 Headroom: the cost of the safety margin

The 15 % headroom in $N_{\text{nodes}}$ amounts to having ~1.6 extra nodes on average. Its annual cost is:

$$C_{\text{headroom}} = 0.15 \times 83{,}300\;\text{EUR/node/year} \times N_{\text{nodes,base}} \approx 12{,}500 \times 9 = 112{,}500\;\text{EUR/year}$$

That cost is justified by:

GPU failure rate ~5 % per year (in small clusters, documented by Introl, Apr. 2026)
Replacement time of 2–8 weeks (depending on market availability)
Unforeseen peaks of up to 20 % above the estimate

If the service can degrade gracefully (relaxing the TTFT SLO from 300 ms to 500 ms at extreme peak), headroom can be cut to 10 %, saving ~37,500 EUR/year.

6. Sensitivity map: €/GPU-hour and break-even on two axes

The table below crosses utilisation with the opex scenario, showing the all-in EUR/GPU-hour (base scenario, capex/year 55,300 €):

	Low opex (13,000 €/year)	Base opex (28,000 €/year)	High opex (75,000 €/year)
Util. 40 %	4.74	5.93	9.06
Util. 55 %	3.45	4.31	6.59
Util. 63 %	3.01	3.76	5.75
Util. 70 %	2.71	3.39	5.18
Util. 80 %	2.37	2.97	4.54
Util. 100 %	1.90	2.38	3.63

The crossing point with the cloud price (AWS OD: 3.63 EUR):

Low-opex scenario: break-even at ~38 % utilisation
Base-opex scenario: break-even at ~41 % utilisation
High-opex scenario: break-even at ~53 % utilisation

The biggest lever for lowering the break-even is not hardware capex but opex (staff and colocation above all). See the analysis of utilisation as a FinOps lever in GPU utilisation as FinOps.

7. Integration with the rest of the FinOps chain

The sizing of §1 determines the number of nodes; the TCO of §2 gives the cost per hour; the comparison of §3 gives the break-even. Those three numbers feed directly into the other instruments of the series:

Instrument	Input from this article	Output
GuideLLM — SLO validation under load	TTFT/ITL P99 SLO from step 2	Experimental confirmation of the real throughput per GPU
Capacity planning for on-premise inference	Demand profile and N GPUs from step 4	Scaling policy and autoscaling triggers
Cost per token and per request	EUR/GPU-hour from step §2.2	EUR/1M tokens per model and batching
Cloud GPU commitment and spot	Cloud prices from table §2.4	Optimisation of the complementary cloud tier
Sovereign on-premise vs hyperscalers	Break-even from §3	Final buy/rent decision including the sovereignty axis
TCO of an on-premise GPU cluster	Capex and opex from §2.1	Detailed TCO model with every line item
GPU utilisation as FinOps	Target utilisation from step 2	Scheduling levers to raise real utilisation

Sources

Spheron · LLM Inference SLO Engineering: TTFT, ITL, and P99 Latency Budgets for Production AI (2026) — https://www.spheron.network/blog/llm-inference-slo-ttft-itl-latency-budget-guide-2026/
Spheron · vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026) — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
Spheron · GPU Cloud Pricing 2026: H100 from 1.03 USD/hr, B200 from 2.12 USD/hr — https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
MLPerf Inference v5.1 — Red Hat: 5 777 tok/s (offline) en Llama 3.1-8B FP8 en H100 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
MLPerf Inference v6.0 Results Explained: GPU Performance Rankings for AI Workloads (2026) — https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
IntuitionLabs · H100 Rental Prices Compared: 1.49–6.98 USD/hr Across 15+ Cloud Providers (2026) — https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison
CloudZero · Cloud GPU Pricing Comparison: AWS vs Azure vs GCP For AI Workloads (2026) — https://www.cloudzero.com/blog/cloud-gpu-pricing-comparison/
Lenovo Press · On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing 2026: Rent vs Buy Cost Analysis — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Red Hat · 233 % 3-year ROI and 13 months to payback with Red Hat AI (feb. 2026) — https://www.redhat.com/en/blog/233-3-year-return-investment-and-13-months-payback-red-hat-ai
VentureBeat · 5 % GPU utilization: the 401 billion USD AI infrastructure problem — https://venturebeat.com/infrastructure/5-gpu-utilization-the-401-billion-ai-infrastructure-problem-enterprises-cant-keep-ignoring/
DZone · Queueing Theory for LLM Inference — https://dzone.com/articles/queueing-theory-for-llm-inference
GuideLLM · Evaluate LLM deployments for real-world inference (Red Hat Developer, jun. 2025) — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference

Full TCO of an on-premise GPU cluster: from capex to the all-in €/GPU-hour and the break-even against cloud

Tue, 16 Jun 2026 06:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator. The dollar sign is not used (it is the formula delimiter). Data centred on Europe/Spain. Generic example hardware: a cluster of N nodes, each with 4×H100 SXM5 80 GB.

TL;DR

A 4×H100 SXM5 node costs between 150,000 USD and 200,000 USD in total capex (GPUs + server + network + storage + prorated rack). Amortised over 3 years with European opex (energy at ~0.116 €/kWh industrial, PUE 1.54 average or 1.2 with liquid, 0.3 FTE of staff), the all-in cost ranges from 3.10 USD/GPU-hour (100 % utilisation) to 6.20 USD/GPU-hour (50 % utilisation). The break-even against AWS p5 on-demand (~6.88 USD/GPU-hour) is crossed at around 70 % utilisation; against a neocloud 3-year reserved (~1.49–2.10 USD/GPU-hour), on-prem never closes the gap in that scenario. Utilisation is the variable that decides the cost axis, not the price of the hardware.

The model: declared assumptions

Every calculation below starts from these assumptions. Changing any of them moves the conclusion; the sensitivity section quantifies by how much.

Parameter	Base value	Sensitivity range
Node	4×H100 SXM5 80 GB (HGX baseboard)	—
Cluster	N nodes (per-node model; scales linearly)	1–32 nodes
Capex amortisation	3 years (straight line)	3–5 years
GPU utilisation	70 %	30 %–100 %
Energy price	0.116 €/kWh (industrial Spain, Sept. 2025)	0.06–0.20 €/kWh
PUE	1.54 (global average, Uptime Institute 2025)	1.15–1.80
Exchange rate	1 USD = 0.93 € (reference Jun. 2026)	—

Energy source: GlobalPetrolPrices · Spain Business Electricity, Sept. 2025. PUE source: Uptime Institute Global Data Center Survey 2025 — the global average PUE has been stuck at 1.54 for the sixth consecutive year; hyperscalers 1.10–1.15; colocation/enterprise 1.58–1.80; facilities less than 5 years old, 1.48. PUE 1.2 is achievable with direct-to-chip liquid cooling.

Capex breakdown per 4×H100 SXM5 node

GPUs

Component	Unit price (USD)	Quantity	Subtotal (USD)	Source and date
H100 SXM5 80 GB (card)	30,000–40,000	4	120,000–160,000	GMI Cloud, Apr. 2026 · Introl, Apr. 2026

The range reflects market variability and volume discounts (5–15 % for orders above 50 units). H100 SXM5 cards require NVIDIA’s HGX baseboard; they are not sold loose for direct installation in standard servers.

Server / HGX baseboard

A complete 4×H100 SXM node uses NVIDIA’s HGX H100 4-GPU baseboard plus a compatible host server. Reference models: Supermicro SYS-421GU-TNXR (4U, dual Intel Xeon 4th Gen, HGX H100 4-GPU) and its Dell equivalent.

Component	Estimated cost (USD)	Note
Server chassis + CPU (2× Xeon) + RAM (512 GB DDR5) + redundant PSU	18,000–25,000	Based on a Supermicro SYS-821GE bare without GPUs at ~24,806 USD (xicomputer.com, Jun. 2026); scaled to the 4-GPU
HGX H100 4-GPU baseboard	included in the GPU price	NVIDIA HGX platform; no separate public price
Inter-GPU NVLink (inside the node)	included in the baseboard	4 GPUs connected by NVLink 4.0 on the HGX baseboard

Marketing claim (no independent verification): Supermicro announces datacenter energy cost reductions of up to 40 % with liquid cooling in its HGX H100 servers (Supermicro press release).

InfiniBand NDR network

For a multi-node cluster with tensor parallelism across nodes, the GPU-to-GPU network is critical. NDR InfiniBand (400 Gb/s per port) is the de facto standard for HGX clusters.

Component	Estimated cost per node (USD)	Source / Note
NVIDIA Quantum-2 NDR 400G switch (64 ports, prorated across N nodes)	2,000–4,000	Switch ~35,000 USD (Introl, Apr. 2026); at 16 nodes, ~2,200 USD/node
InfiniBand NDR cables/transceivers (4 ports per node × ~1,000 USD/port)	4,000	Estimate based on ~1,000 USD per optical transceiver (Introl, Apr. 2026)
InfiniBand network (prorated per 4-GPU node)	~6,000–8,000	—

For inference serving inside a single node (4 GPUs with NVLink), the inter-node network is less critical than for multi-node training. For disaggregated prefill-decode workloads spread across nodes, InfiniBand NDR is necessary.

NVMe storage

Component	Estimated cost (USD)	Note
Local NVMe (4 TB × 2 U.2/E1.S drives, working datasets and checkpoints)	2,000–4,000	~500–1,000 USD/TB enterprise NVMe, 2025
Shared object storage (NAS/MinIO, prorated per node)	2,000–5,000	Varies with the total capacity of the cluster
Total storage per node	~4,000–9,000	—

Introl models 50 TB per GPU for effective operation in training clusters (Introl, Apr. 2026); for pure inference the requirement is significantly lower (model weights + logs).

Rack, PDU and datacenter connectivity

Component	Estimated cost per node (USD/year)	Source
Rack colocation (high density, 10–15 kW per node)	5,000–12,000/year	Encoradvisors · Colocation Pricing 2026: high density 3,000–6,000 USD/month per rack; at 2 nodes per rack, ~1,500–3,000 USD/month per node = 18,000–36,000 USD/year in tier-1; lower in Spain
Rack PDU, electrical cabling (prorated)	500–1,000 per node (amortised capex)	Inside the colocation line item or in an own datacenter

Colocation in Spain/Europe is structurally cheaper than in US tier-1 markets (New York, Silicon Valley). For an own datacenter, replace it with the cost of your own space plus amortisation of the electrical and cooling infrastructure.

Capex summary per 4×H100 SXM5 node

Line item	Range (USD)	Midpoint
GPUs (4× H100 SXM5)	120,000–160,000	140,000
Server chassis + CPU/RAM/PSU	18,000–25,000	21,500
InfiniBand NDR network (prorated)	6,000–8,000	7,000
NVMe + object storage	4,000–9,000	6,500
PDU/rack/other (capex)	2,000–5,000	3,500
Total capex per node	150,000–207,000	178,500

Sources: GMI Cloud (Apr. 2026), Introl (Apr. 2026), Spheron (Apr. 2026), xicomputer.com (Jun. 2026).

Opex breakdown per 4×H100 SXM5 node (annual)

Energy

A 4×H100 SXM5 node at full load draws approximately:

$$P_{\text{node}} = 4 \times 700\,\text{W (TDP H100 SXM5)} + 800\,\text{W (server)} \approx 3.6\,\text{kW (IT)}$$

The total datacenter power includes the cooling overhead, expressed by the PUE:

$$P_{\text{total}} = P_{\text{IT}} \times \text{PUE}$$ $$\text{annual energy cost} = P_{\text{IT}} \times \text{PUE} \times 8{,}760\,\text{h} \times \text{kWh price}$$

With the base values (PUE 1.54; 0.116 €/kWh):

$$\text{energy/year} = 3.6\,\text{kW} \times 1.54 \times 8{,}760\,\text{h} \times 0.116\,\text{EUR/kWh} \approx 5{,}475\,\text{EUR}$$

With a Spanish solar PPA (Q3 2025 reference price: ~34 €/MWh = 0.034 €/kWh according to PV Tech, Oct. 2025):

$$\text{energy/year (solar PPA)} = 3.6 \times 1.54 \times 8{,}760 \times 0.034 \approx 1{,}604\,\text{EUR}$$

Energy scenario	Price (€/kWh)	Energy cost/year per 4-GPU node
Spanish solar PPA (Q3 2025)	0.034	~1,604 €
Industrial Spain (Sept. 2025)	0.116	~5,475 €
European average (industrial tariff)	0.160	~7,550 €
Worst case (no PPA, high tariff)	0.200	~9,437 €

Staff / operations

Staff cost is the most variable line item with cluster size. For a small cluster (2–8 nodes), the rule of thumb is 0.3–0.5 FTE per cluster of GPU infrastructure support (Spheron, Apr. 2026).

Cluster size	Estimated FTE	FTE cost (€/year, Western Europe)	Cost per 4-GPU node (€/year)
2–4 nodes	0.3 FTE	~120,000	36,000–18,000
8–16 nodes	0.5 FTE	~120,000	7,500
32+ nodes	1–2 FTE	~120,000	3,750–7,500

Indicative salary reference: a GPU infrastructure engineer with CUDA, InfiniBand and Kubernetes knowledge in Western Europe, ~90,000–140,000 €/year fully loaded. Introl’s figures (Apr. 2026) in USD (~275,000 USD/year for the US) reflect the North American market, which is appreciably higher.

Maintenance and support

Line item	Annual cost (% of hardware capex)	Per 4-GPU node (midpoint)
Vendor maintenance / support	5–10 % of capex	~7,000–14,000 USD → ~6,500–13,000 €
GPU failure rate (~5 % annual) × replacement cost	5 % × 4 GPUs × ~35,000 USD = ~7,000 USD expected	~6,500 € (amortised as a provision)
Minor spares (cables, modules)	~500–1,000 €	—

Introl quotes GPU failure rates of 2–3 % per year in small clusters; Google Research documented ~9 % annualised in Meta’s 16,384-GPU H100 cluster (Introl, Apr. 2026). 5 % is used here as a conservative intermediate value.

Depreciation (for accounting purposes)

Straight-line depreciation turns capex into an annual flow comparable to the cost of committed cloud:

$$\text{annual depreciation} = \frac{\text{node capex}}{\text{amortisation years}}$$

Node capex (USD)	3-year amortisation (USD/year)	5-year amortisation (USD/year)
150,000 (minimum)	50,000	30,000
178,500 (average)	59,500	35,700
207,000 (maximum)	69,000	41,400

H100 hardware depreciates fast: secondary-market analyses put the residual value at 20–40 % of the purchase price after 3 years (Introl, Apr. 2026). The arrival of Blackwell GB200/GB300 accelerates the perceived obsolescence.

Annual opex summary per 4×H100 SXM5 node (base scenario, 8-node cluster)

Line item	Base scenario (€/year)	Range
Energy (PUE 1.54; 0.116 €/kWh)	5,475	1,604–9,437
Staff (0.5 FTE × 8 nodes, prorated)	7,500	3,750–36,000
Maintenance / support / failures	9,000	5,000–15,000
Rack colocation (Spain, high density)	6,000	3,000–15,000
Total opex per node	~28,000	~13,000–75,000

The extreme range reflects the difference between a well amortised own datacenter with a solar PPA and cheap energy (minimum opex) and tier-1 colocation with market tariffs and senior staff.

Deriving the all-in €/GPU-hour

Formula

$$\text{EUR/GPU-hour all-in} = \frac{\frac{\text{node capex}}{\text{years}} + \text{node annual opex}}{4\,\text{GPUs} \times 8{,}760\,\text{h} \times u}$$

where $u$ is the average annual utilisation (0 to 1).

See the cost-per-token identity in cost per token and per request for the connection with throughput.

€/GPU-hour table by utilisation and scenario

Average capex (178,500 USD → ~166,000 €), 3-year amortisation → 55,300 €/year.

Utilisation	Opex/year (base, €)	Total cost/year (€)	Useful GPU-hours/year	€/GPU-hour
30 %	28,000	83,300	10,512	7.93
50 %	28,000	83,300	17,520	4.75
70 %	28,000	83,300	24,528	3.39
80 %	28,000	83,300	28,032	2.97
100 %	28,000	83,300	35,040	2.38

Low-opex scenario (solar PPA, own datacenter, large cluster): opex/year ~13,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	68,300	3.90
70 %	68,300	2.78
80 %	68,300	2.43
100 %	68,300	1.95

High-opex scenario (market tariff, expensive colocation, small cluster): opex/year ~75,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	130,300	7.44
70 %	130,300	5.31
80 %	130,300	4.65
100 %	130,300	3.72

From €/GPU-hour to €/1M tokens

The cost-per-token identity connects the hardware cost with the inference cost:

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^6}{\text{throughput (tok/s)} \times 3{,}600}$$

For reference throughputs on H100 SXM5 with vLLM (see capacity planning for on-premise inference):

Model	Typical throughput (tok/s per GPU)	Source
Llama-3 70B FP8, high batch	~2,800	Series B benchmarks
Llama-3 8B FP16, medium batch	~9,000	Series B benchmarks
Mixtral 8×7B, high batch	~4,500	Series B benchmarks

€/1M tokens table in the base scenario (€/GPU-hour 3.39 at 70 % utilisation):

Model	Throughput (tok/s)	€/1M tokens
Llama-3 70B FP8	2,800	~0.336
Llama-3 8B FP16	9,000	~0.105
Mixtral 8×7B	4,500	~0.209

At 50 % utilisation (€/GPU-hour 4.75):

Model	€/1M tokens
Llama-3 70B FP8	~0.471
Llama-3 8B FP16	~0.147

Occupancy (batching) multiplies the effective throughput and lowers the €/1M tokens without changing the hardware; it is analysed in GPU utilisation as a FinOps lever.

Break-even on-prem vs cloud

The break-even formula

Break-even happens when the total annual on-prem cost equals the annual cloud cost at the same utilisation:

$$\text{annual cloud cost} = \text{cloud GPU-hour price} \times 4\,\text{GPUs} \times 8{,}760\,\text{h} \times u$$ $$\text{break-even}: \quad \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u} = \text{cloud GPU-hour price}$$

Solving for the break-even utilisation:

$$u^* = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times \text{cloud GPU-hour price}}$$

Break-even table by cloud mode and on-prem scenario

Base on-prem scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/year per 4-GPU node):

Cloud reference (price/GPU-hour)	USD equiv.	Break-even utilisation $u^*$	Note
Neocloud on-demand (Lambda/Spheron ~2.90 USD)	2.90 USD (~2.70 €)	>100 %, on-prem does not compete	Neocloud on-demand is cheaper even at full utilisation
Neocloud 3-year reserved (CoreWeave ~1.49–2.10 USD)	~1.80 USD (~1.67 €)	>100 %, impossible	Neocloud reserved beats on-prem in every scenario of this model
AWS p5 on-demand (6.88 USD/GPU-hour)	6.88 USD (~6.40 €)	~47 %	Above 47 %, the average on-prem beats AWS on-demand
AWS p5 3-year reserved (~2.97 USD/GPU-hour)	2.97 USD (~2.76 €)	>100 %	—
GCP A3 on-demand (~10.98 USD/GPU-hour)	10.98 USD (~10.21 €)	~29 %	Above 29 %, on-prem beats GCP on-demand
Azure ND H100 v5 on-demand (~12.29 USD/GPU-hour)	12.29 USD (~11.43 €)	~26 %	—

Low-opex scenario (total 68,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~38 %
Neocloud on-demand (2.90 USD ≈ 2.70 €)	~91 %
Neocloud 3-yr reserved (1.80 USD ≈ 1.67 €)	>100 %

High-opex scenario (total 130,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~72 %
GCP A3 on-demand (~10.21 €)	~45 %
Azure on-demand (~11.43 €)	~41 %

Reading the break-even table

Against neoclouds (on-demand or reserved), on-prem TCO does not reach break-even in any scenario of the base model. Neocloud reserved beats on-prem even at 100 % utilisation, because its hourly price is below the all-in cost of your own hardware. This is consistent with the analysis in cloud GPU: on-demand, reserved and spot.
Against on-demand hyperscalers (AWS, GCP, Azure), on-prem does have an achievable break-even: around 26–72 % utilisation depending on the scenario. At medium-high utilisation (>70 %), on-prem clearly beats AWS/GCP/Azure on-demand.
The variable that moves the break-even most is opex (staff above all), not hardware capex. A well sized cluster in cheap colocation with PPA energy can lower the threshold by 20 percentage points compared with the high scenario.
For GDPR data, the break-even against US hyperscalers is skewed: the sovereignty axis rules out US hyperscalers before cost does (see sovereign on-premise vs hyperscalers).

Sensitivity analysis

TCO vs utilisation

The all-in cost per GPU-hour varies inversely with utilisation because capex is fixed:

$$\frac{d(\text{EUR/GPU-hour})}{du} = -\frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u^2} < 0$$

Going from 50 % to 80 % utilisation cuts the €/GPU-hour by $\frac{4.75 - 2.97}{4.75} \approx 37\,\%$ in the base scenario. That 37 % reduction requires no hardware change, only more efficient scheduling (see GPU utilisation as a FinOps lever).

Utilisation	€/GPU-hour (base scenario)	Change vs 50 %
30 %	7.93	+67 %
50 %	4.75	reference
70 %	3.39	−29 %
80 %	2.97	−37 %
100 %	2.38	−50 %

TCO vs energy price

Energy price (€/kWh)	Energy opex/year	€/GPU-hour (70 % util.)	Change vs base
0.034 (solar PPA)	1,604 €	3.00	−12 %
0.116 (industrial ES, base)	5,475 €	3.39	reference
0.160 (European average)	7,550 €	3.54	+4 %
0.200 (high tariff)	9,437 €	3.67	+8 %

Energy has a moderate impact on total TCO (8–12 % variation between the extremes), because hardware capex dominates. Over a very long amortisation (5 years) with a solar PPA, however, energy drops from 6 % to 1 % of total TCO and the differential is amplified. The price of energy matters more for the carbon footprint (CSRD) than for TCO when capex is dominant.

TCO vs PUE

PUE	Cooling overhead	Energy/year (0.116 €/kWh)	€/GPU-hour (70 % util.)
1.15 (liquid cooling, new facilities)	+15 %	2,166 €	3.21
1.20 (liquid, modern datacenter)	+20 %	2,259 €	3.23
1.48 (facilities <5 years, Uptime 2025)	+48 %	3,490 €	3.33
1.54 (global average, Uptime 2025)	+54 %	3,627 €	3.39
1.80 (legacy colocation)	+80 %	4,260 €	3.47

The difference between PUE 1.15 (liquid) and 1.80 (legacy) is barely ~8 % of the €/GPU-hour at 70 % utilisation, because energy is only a fraction of TCO. PUE matters far more for the absolute energy cost and for CSRD reporting than for total TCO when hardware is the dominant component.

TCO vs amortisation years

Amortisation	Capex/year (average node, USD)	€/GPU-hour (70 % util., base opex scenario)
3 years	59,500 USD (~55,300 €)	3.39
4 years	44,625 USD (~41,500 €)	2.99
5 years	35,700 USD (~33,200 €)	2.72

Stretching the amortisation from 3 to 5 years lowers the €/GPU-hour by ~20 %, assuming the hardware remains competitive and the resale market supports the residual value. With the refresh cycle accelerated by Blackwell GB200/GB300, a 5-year amortisation carries a higher risk of technological obsolescence.

Sensitivity heat map (€/GPU-hour at 70 % utilisation, base scenario)

	PUE 1.15	PUE 1.54	PUE 1.80
3-yr amort., solar PPA (0.034 €)	2.72	2.74	2.76
3-yr amort., industrial (0.116 €)	3.21	3.39	3.47
5-yr amort., industrial (0.116 €)	2.54	2.72	2.80
3-yr amort., high tariff (0.200 €)	3.44	3.67	3.78

Decision table: cost/control/sovereignty Pareto

The table below crosses the four dimensions with no implicit hierarchy; the ordinal reading depends on each organisation’s constraints.

Option	€/GPU-hour	Initial capex	Full stack control	EU sovereignty	Elasticity	Operational risk
On-prem (util. >70 %, low opex)	2.40–3.00	high (150–207k USD/node)	total	total	none	hardware failure, idle
On-prem (util. <50 %, base opex)	4.75–7.93	high	total	total	none	capex with no return
Neocloud 3-year reserved (CoreWeave, Lambda)	1.49–2.10 USD	none	partial (API)	depends on the provider	rigid contract	minimal interruption
Neocloud on-demand (Lambda, Spheron)	2.49–3.44 USD	none	partial	depends	total	no interruption
AWS p5 on-demand	6.88 USD	none	minimal	NO (CLOUD Act)	total	no interruption
AWS p5 3-year reserved	~2.97 USD	financial commitment	minimal	NO (CLOUD Act)	rigid	no interruption
Sovereign EU cloud (Scaleway, Nebius EU)	2.15–3.85 USD	none	partial	yes (EU)	total	no interruption
Hybrid on-prem base + EU cloud peak	2.00–3.50 (weighted)	medium	high	yes (EU)	elastic peak	operational complexity

“EU sovereignty” column: US hyperscalers (AWS, GCP, Azure) are subject to the US CLOUD Act regardless of the datacenter region. Nebius has a Dutch legal entity; CoreWeave is a US company. See the full analysis in sovereign on-premise vs hyperscalers.

“Full stack control” column: on-prem lets you choose the driver version, the kernel, the NCCL configuration, MIG partitioning, and any system parameter. Cloud options offer control at container/pod level, with the hypervisor and firmware opaque.

The cost/sovereignty Pareto frontier for GDPR data excludes US hyperscalers, leaving: on-prem, sovereign EU cloud, and the hybrid. Among those three, the deciding variable is sustained utilisation and the predictability of traffic (see capacity planning for on-premise LLM inference).

Integration with the series FinOps model

The all-in €/GPU-hour of on-prem is the number that feeds the cost allocation pipeline of the series:

Cost-per-token identity (cost per token and per request): engine throughput × €/GPU-hour → €/1M tokens.
Chargeback and showback (chargeback and showback in GPU multitenancy): the all-in €/GPU-hour is the internal price charged to each tenant of the multi-tenant cluster.
Utilisation as a lever (GPU utilisation as FinOps): raising utilisation from 50 % to 80 % cuts the €/GPU-hour by 37 % without changing the hardware, the highest ROI in on-prem FinOps.
Capacity planning (capacity planning for on-premise LLM inference): the number of nodes to buy depends on the percentile of base load you want to cover in iron.
Cloud comparison (cloud GPU: on-demand, reserved and spot): the all-in €/GPU-hour goes head to head with the cloud price in table A7 to compute the break-even.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis (abr. 2026) — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis (abr. 2026) — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Uptime Institute · Global Data Center Survey 2025 (PDF oficial) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf
Uptime Institute · Global Data Center PUE Stalls at 1.54 (comunicado, oct. 2025) — https://mgrid.org/2025/10/01/uptime-institute-data-center-pue-stagnation-2025-liquid-cooling/
GlobalPetrolPrices · Spain Business Electricity Price (sept. 2025) — https://www.globalpetrolprices.com/Spain/electricity_prices/
PV Tech · European Solar PPA Prices Fall Below 35 €/MWh in Q3 2025 (oct. 2025) — https://www.pv-tech.org/european-solar-ppa-prices-fall-below-35-mwh-q3-2025/
Xi Computers · Supermicro SYS-821GE-TNHR 8U GPU Server Pricing (jun. 2026) — https://www.xicomputer.com/Solutions/Data-Center-Servers/SYS-821GE-TNHR.asp
Encoradvisors · Data Center Colocation Pricing 2026 — https://encoradvisors.com/data-center-colocation-pricing/
CloudZero · H100 GPU Cost In 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/
Spheron · AI Inference Power Consumption and GPU Electricity Costs: 2026 Guide — https://www.spheron.network/blog/ai-inference-power-electricity-cost-2026/
NVIDIA · DGX SuperPOD H100 Electrical Specifications (documentación oficial) — https://docs.nvidia.com/dgx-superpod/design-guides/dgx-superpod-data-center-design-h100/latest/electrical.html

Cloud GPU: on-demand, reserved and spot, and why neoclouds change the board (with 2026 prices)

Tue, 16 Jun 2026 05:00:00 +0200

Notation: amounts in N USD or N €, decimal point. The dollar sign is not used (on this site it is the formula delimiter). All cloud prices change without notice; the date of the source is given next to each figure.

TL;DR

An H100 SXM5 on-demand costs between 2.49 USD/GPU-hour (Lambda, a neocloud) and 12.29 USD/GPU-hour (Azure ND H100 v5) depending on the provider, a spread of 5× between the neocloud floor and the hyperscaler ceiling. The reserved mode (a 1-3 year commitment) lowers the price by 20-40 % and is the number comparable to amortising your own iron; the spot/preemptible mode takes another 40-65 % off, but with a risk of interruption on less than 2 minutes’ notice. For the on-prem vs cloud analysis of this series (article sovereign on-premise vs hyperscalers), neocloud reserved is the correct reference: at 1.49-2.10 USD/GPU-hour with a 3-year commitment, neoclouds compete head-on with the TCO of your own iron at medium utilisation.

The three cloud GPU purchasing models

On-demand

Pay per use with no commitment, billed by the hour or by the minute. Maximum flexibility; maximum price. Applies to: development, experimentation, occasional capacity peaks and any workload whose lifetime is unpredictable. The absence of commitment has a cost: the on-demand premium over reserved ranges from 25 % to 60 % depending on the provider.

Reserved / committed

A 1 or 3 year contract that guarantees capacity in exchange for a discount on the on-demand rate. At hyperscalers it goes by the name Reserved Instance (AWS), Committed Use Discount (GCP) or Azure Reservation. At neoclouds it is negotiated directly and can include the choice of node, network (Ethernet or InfiniBand) and region. Typical discounts:

Term	Discount on on-demand (market reference)
1 year	20-37 %
3 years	30-45 %

Reserved is the model comparable to on-prem: in both cases a fixed cost is incurred up front (amortised capex on-prem; committed payment in the cloud). The formula for the effective cost per GPU-hour is the same in both cases:

$$\text{cost/GPU-hour}_{reserved} = \frac{\text{committed price (USD/GPU-h)} \times 8760}{\text{contracted hours}}$$

Unlike on-prem, cloud reserved includes neither hardware capex nor operational opex, but neither does it include the benefits of running your own iron at high load (see GPU utilisation as a FinOps lever).

Spot / preemptible

Idle provider capacity offered at a 40-65 % discount on on-demand, on the condition that it can be reclaimed with 30 seconds to 2 minutes of notice (AWS Spot, GCP Spot VMs). The discount is real and reproducible; so is the interruption risk.

Workloads suitable for spot:

Workload	Requires checkpointing	Suitable for spot
LLM pre-training (checkpoint-resume every 15-30 min)	yes	yes
LoRA/QLoRA fine-tuning with checkpoints	yes	yes
Offline batch inference (embeddings, evaluations)	no (idempotent)	yes
Hyperparameter search	no (repeatable)	yes
Production serving with a latency SLO	not applicable	no
Production serving with an availability SLO	not applicable	no

A fine-tune of a 7B model on 4× H100 over 40 hours on spot achieved real savings of 81 % against the equivalent on-demand (Spheron Blog, May 2026). The typical interruption rate on high-reliability H100 instances is 2-8 % per day, which gives a 92-98 % probability of completing a 24-hour shift without a cut (Thunder Compute, Jun. 2026).

The neoclouds: GPU-specialised clouds

Definition and differentiators

Neoclouds (or GPU-specialised clouds) are infrastructure providers focused exclusively on GPU compute for AI workloads, without the diversity of services of the hyperscalers. SemiAnalysis coined the term in 2024 and identifies CoreWeave, Nebius, Lambda and Crusoe at the top tier (SemiAnalysis, 2024).

Structural differences with respect to the hyperscalers:

Dimension	Neoclouds (CoreWeave, Lambda, Nebius…)	Hyperscalers (AWS, GCP, Azure)
H100 on-demand price	2.49-6.16 USD/GPU-hour	6.88-12.29 USD/GPU-hour
Interconnect network	InfiniBand NDR 400 Gbps (CoreWeave, Voltage Park), Ethernet 200+ Gbps (Lambda, Nebius)	Proprietary Ethernet (EFA, GVNIC) or InfiniBand (only p5/H100 on AWS)
Access to new hardware	NVIDIA priority access (CoreWeave is an NVIDIA shareholder); H200/B200 catalogue available earlier	Slower catalogue; B200 in reserved or limited capacity in 2026
Contracts	1-36 months, negotiated directly	Standard RI (1-3 years), no node negotiation
Additional services	Managed Kubernetes, storage, bare-metal	Complete ecosystem (IAM, VPC, databases, ML services…)
Egress	Zero-egress or flat fee (Lambda, Voltage Park, Nebius)	0.08-0.12 USD/GB (AWS, GCP, Azure)
Sovereignty	Varies: Nebius (EU), Crusoe (US, clean), CoreWeave (US/EU)	US CLOUD Act; no guarantee of EU jurisdiction

The neocloud market is estimated at 20,000 M USD of revenue in 2026 and is growing towards 180,000 M USD in 2030 (Synergy Research Group, 2026). CoreWeave passed 5,000 M USD of ARR faster than any other cloud platform. Neoclouds price their H100s 60-85 % below AWS and Azure for the same silicon (Signisys, 2026).

Profiles of the main neoclouds

Provider	Main hardware	Network	Jurisdiction	Relevant note
CoreWeave	H100, H200, B200, GB200	InfiniBand NDR (Quantum-2 SHARP)	US / EU	NVIDIA shareholder; priority access to hardware; largest neocloud by fleet
Lambda Labs	H100, A100, B200	Ethernet	US	Zero-egress; no spot; per-minute billing
Nebius	H100, H200, B200, B300	InfiniBand	EU (Amsterdam)	Spin-off of Yandex Cloud; zero-egress; object egress 0.015 USD/GiB; public on-demand price
Crusoe	H100, A100, MI300X	Ethernet	US	100 % renewable/flare gas energy; AMD MI300X at 3.45 USD/hr
Voltage Park	H100	Ethernet (1.99 USD) / InfiniBand 3200 Gbps (2.49 USD)	US	Merged with Lightning AI in Jan. 2026; no minimums; non-profit Navigation Fund
Together AI	H100, A100	Ethernet	US	Inference platform + raw compute
GMI Cloud	H100, H200	Ethernet	Asia/US	H200 on-demand from 2.60 USD/hr
RunPod	H100, H200, B200, A100	Ethernet / Secure Cloud	Global	Community + Secure Cloud with SLA; spot available

Note: the “zero-egress”, “renewable” and priority hardware access claims are marketing from each provider and have not been verified by independent audits. They are reproduced with that caveat.

2026 price table: H100 SXM by provider and mode

Source and date given for each figure. Prices change without notice.

H100 SXM5 (80 GB HBM3) — USD per GPU-hour

Provider	On-demand	1-year reserved	3-year reserved	Spot/Preemptible	Source and date
CoreWeave	~6.16	~2.10-3.00 (negotiated)	~1.49-2.10 (negotiated)	not available	Thunder Compute, Jun. 2026
Lambda Labs	2.49-3.44	~1.89 (37 % disc.)	~1.84 (PCIe 3-yr)	not available	Spheron Blog, May 2026
Nebius	3.85	discount up to 35 % (contact)	discount up to 35 % (contact)	2.15 (preemptible)	nebius.com/prices, Jun. 2026
Voltage Park	1.99 (Ethernet) / 2.49 (InfiniBand)	contact (≥6 months)	contact	not available	voltagepark.com/pricing, Jun. 2026
RunPod (Secure)	3.29	not available	not available	available (~59 % disc.)	Spheron Blog, May 2026
Spheron	2.50	volume discount (contact)	volume discount (contact)	1.03	Spheron pricing, May 2026
AWS (p5.48xlarge)	6.88	~2.97 (3-yr, ~23.78 USD/h node ÷ 8)	~2.97	~1.14 (p5 spot, 9.10 USD/h node)	Vantage, Jun. 2026
GCP (A3 High)	10.98	~8.78 (1-yr CUD)	n/a	~3.69	Spheron Blog, May 2026
Azure (ND H100 v5)	12.29	~6.15-8.60 (1-3 yr reservation)	~6.15-8.60	~2.27 (node spot ÷ 8)	Vantage Azure, Jun. 2026

All amounts in USD/GPU-hour, rounded to 2 decimals. The “3-year reserved” column is the number comparable to the amortisation cost of your own hardware.

AWS p5.48xlarge: an 8× H100 node; on-demand 55.04 USD/h → 6.88 USD/GPU-h; 3-yr reserved 23.78 USD/h → 2.97 USD/GPU-h; spot 9.10-25.35 USD/h (varies by region and availability). GCP a3-highgpu-8g: 8× H100 SXM5; on-demand 87.84 USD/h → 10.98 USD/GPU-h; spot ~3.69 USD/GPU-h (us-central1, May 2026). Azure ND96isr H100 v5: 8× H100; on-demand 98.32 USD/h → 12.29 USD/GPU-h; node spot 18.17 USD/h → ~2.27 USD/GPU-h (source: Vantage, Jun. 2026, us-east-1).

2026 price table: H200 and B200

H200 SXM (141 GB HBM3e) — USD per GPU-hour

Provider	On-demand	Spot/Preemptible	Source and date
Nebius	4.50	2.45 (preemptible)	nebius.com/prices, Jun. 2026
GMI Cloud	2.60	not available	GMI Cloud, May 2026
RunPod	4.39	not available	Spheron Blog, May 2026
Spheron	4.54	not available	Spheron pricing, May 2026
CoreWeave	6.31	not available	GMI Cloud, May 2026
AWS (p5e)	~4.98	limited	Spheron Blog, May 2026
Azure	~13.78	not available	Spheron Blog, May 2026

B200 SXM6 (192 GB HBM3e) — USD per GPU-hour

Provider	On-demand	Spot	Source and date
Lambda Labs	4.99-5.29	not available	Spheron Blog, May 2026
Nebius (HGX B200)	7.15	3.95 (preemptible)	nebius.com/prices, Jun. 2026
RunPod	5.89	not available	Spheron Blog, May 2026
Spheron	6.02	2.12	Spheron pricing, May 2026
AWS (p6-B200)	~14.24	~3.24	Spheron Blog, May 2026

B200 availability is still limited in Q2 2026; many providers offer only access by advance reservation. The prices are market estimates subject to rapid change.

How it fits into the on-prem vs cloud comparison

The cloud cost per GPU-hour is the number that goes head to head with the on-prem TCO computed in sovereign on-premise vs hyperscalers. The relationship between the three purchasing modes and the cost analysis is as follows:

Cloud mode	What it represents in the comparison	On-prem equivalent
On-demand	marginal cost of peak capacity; floor of build-vs-buy at 100 % utilisation	— (not comparable: on-prem has no elasticity)
3-year reserved	the committed cost comparable to capex amortised over 3 years	node amortisation + fixed annual opex
Spot	the cloud price floor, but with interruption risk	— (not comparable: on-prem is not interrupted)

The cost per token formula applies the same to cloud as to on-prem:

$$\text{cost/1M tokens}_{cloud} = \frac{\text{GPU-hour price (USD)} \times 10^6}{\text{throughput (tok/s)} \times 3600}$$

With an H100 SXM5 at 2,800 tok/s (typical throughput for a 70B model in FP8):

Mode	Reference price	Cost/1M tokens (70B, 2800 tok/s)
Neocloud on-demand (Lambda)	2.49 USD/GPU-h	~0.247 USD (~0.23 €)
Neocloud 3-yr reserved (CoreWeave)	~1.49 USD/GPU-h	~0.148 USD (~0.14 €)
Neocloud spot (Spheron/Nebius)	~1.03-2.15 USD/GPU-h	~0.102-0.213 USD
AWS on-demand (p5)	6.88 USD/GPU-h	~0.682 USD (~0.63 €)
AWS 3-yr reserved (p5)	~2.97 USD/GPU-h	~0.295 USD (~0.27 €)
GCP spot (A3)	3.69 USD/GPU-h	~0.366 USD (~0.34 €)
On-prem amortised, high util. (80 %)	~2.83 USD/GPU-h all-in	~0.281 USD (~0.26 €)

Reference exchange rate: 1 USD = 0.924 €. The rate fluctuates; comparative calculations must be done in the same currency.

The key reading: at high utilisation, on-prem TCO (~2.83 USD/GPU-hour all-in according to Spheron, 2026) competes with the 3-year reserved of the neocloud (~1.49-2.97 USD/GPU-hour). On-prem wins at high utilisation when the all-in cost drops below 2.5 USD; neocloud reserved wins when utilisation is medium or low, because it does not pay for idle capex. Spot lowers the cloud floor but is not comparable: interruption makes it incompatible with continuous serving, which is the case where on-prem competes.

Network factors: InfiniBand vs Ethernet

For distributed training and large-scale inference (tensor parallelism across nodes), the bandwidth of the interconnect network between GPUs is critical. Neoclouds differ in what they offer:

Provider	GPU-GPU network	Bandwidth	Applies to
CoreWeave	InfiniBand NDR (Quantum-2, SHARP)	400 Gbps/GPU	multi-node HGX clusters
Voltage Park	InfiniBand Quantum-2	3200 Gbps (8-GPU node)	the 2.49 USD/hr tier
Voltage Park	Ethernet	200 Gbps	the 1.99 USD/hr tier
Lambda Labs	Ethernet	200 Gbps	all nodes
Nebius	InfiniBand	400 Gbps/GPU	HGX H100/H200/B200
AWS p5	EFA (Elastic Fabric Adapter)	3200 Gbps (node)	EC2 clusters
GCP A3	GVNIC + RoCE	200 Gbps/GPU	A3 High
Azure ND H100 v5	InfiniBand NDR	400 Gbps/GPU	ND96isr H100 v5

InfiniBand NDR with SHARP (in-network collective reduction) cuts the latency of allreduce operations in distributed training. For inference serving with tensor parallelism inside a node (NVLink), the inter-node network is less critical; for multi-node training or disaggregated prefill-decode (the three axes of the series), the network between nodes matters.

Neoclouds and sovereignty: what changes

Unlike the US hyperscalers subject to the US CLOUD Act, some neoclouds offer options with jurisdiction outside the US:

Provider	EU region	US CLOUD Act applies
Nebius	Amsterdam (NL), Frankfurt (DE)	No (Dutch legal entity)
CoreWeave	Frankfurt (DE), London (GB)	Yes (US company)
Lambda Labs	US only	Yes
Voltage Park	US only	Yes
Scaleway	Paris (FR), Amsterdam (NL)	No (French company)
US hyperscalers	EU regions available	Yes (CLOUD Act applies)

For data subject to GDPR, only providers with an EU/EFTA legal entity guarantee the absence of the CLOUD Act. The full discussion of the sovereignty axis is in sovereign on-premise vs hyperscalers. For capacity planning with this data, see capacity planning for on-premise inference.

Decision table: purchasing mode by cost/risk/sovereignty axis

Mode	Cost	Operational risk	Sovereignty (EU)	When to use
Neocloud on-demand	medium (2-4 USD/GPU-h)	low (no interruption)	depends on the provider	development, unpredictable peaks, serving with an SLO
Neocloud 1-yr reserved	low-medium (1.5-3 USD/GPU-h)	low (no interruption, guaranteed capacity)	depends on the provider	stable serving, scheduled training; validated workload
Neocloud 3-yr reserved	low (1.5-2.1 USD/GPU-h)	low, financial commitment	depends on the provider	comparable to amortised on-prem; very stable base load
Neocloud spot	minimum (1-2.5 USD/GPU-h)	high (interruption <2 min)	depends on the provider	batch, fine-tuning with checkpoints, hyperparameter search
Hyperscaler on-demand	high (6.9-12.3 USD/GPU-h)	low	NO (CLOUD Act)	integration with AWS/GCP/Azure services; no alternative
Hyperscaler reserved	medium-high (3-9 USD/GPU-h)	low	NO (CLOUD Act)	lock-in to the hyperscaler ecosystem is justified
Hyperscaler spot	medium (1.1-3.7 USD/GPU-h)	high (interruption)	NO (CLOUD Act)	batch with no sovereignty requirement; more availability than neocloud spot
Amortised on-prem (>70 % util.)	low (~2.3-2.8 USD/GPU-h all-in)	low (no interruption)	total (your own hardware)	stable, high base load; maximum sovereignty

On-prem competes with neocloud 3-yr reserved when utilisation exceeds 65-70 % (see GPU utilisation as FinOps).

Computing the spot saving: an example with checkpointing

Reference parameters to validate whether spot is worth it:

$$\text{spot saving} = (\text{OD price} - \text{spot price}) \times \text{total GPU-hours} - \text{retry cost}$$ $$\text{retry cost} = \text{interruption rate} \times \text{lost work (GPU-hours)} \times \text{spot price}$$

With checkpointing every 30 minutes, work lost per interruption is ≤ 0.5 GPU-hours per event. At an interruption rate of 4 % per day on one H100 GPU:

Item	Reference value
On-demand price (neocloud)	2.50 USD/GPU-h
Spot price	1.03 USD/GPU-h
Gross saving per GPU-hour	1.47 USD
Expected interruptions in 40 h (4 %/day)	~0.67 events
Re-run work per event (0.5 h × 1.03 USD)	~0.34 USD
Net saving on a 40 h × 4 GPU job	~234 USD (against 400 USD on-demand)

The 58 % saving only materialises if the job implements checkpoint-resume correctly. Without checkpoints, a single interruption forces a restart from scratch: the saving turns into an overcost from duplicated work.

Integration into the series TCO model

The cloud €/GPU-hour is the number that feeds the FinOps analysis of the series. The chain:

Cost of cloud iron (this price table) → equivalent to amortised capex + opex of on-prem.
Throughput (measured with genai-perf, track B of the series) → the denominator.
Cost per token (cost per token and per request) → the final comparative metric.
Utilisation (GPU utilisation as FinOps) → in on-demand cloud, utilisation does not penalise cost (you pay for what you use); in reserved and in on-prem it does penalise, because the cost is fixed.

The correct comparison is neocloud 3-yr reserved vs on-prem amortised over 3 years, with the same assumptions on throughput, egress and utilisation. Spot is the cloud price floor but not the comparable floor: its interruptible nature excludes it from any comparison with an on-prem cluster that serves continuous traffic.

Sources

Spheron Blog · GPU Cloud Pricing 2026: H100 from 1,03 USD/hr, B200 from 2,12 USD/hr (15+ providers), may. 2026 — https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
Vantage · p5.48xlarge pricing and specs (AWS), actualizado jun. 2026 — https://instances.vantage.sh/aws/ec2/p5.48xlarge
Nebius · NVIDIA GPU Pricing (página oficial, jun. 2026) — https://nebius.com/prices
Voltage Park · H100 GPU Pricing (página oficial, jun. 2026) — https://www.voltagepark.com/pricing
Thunder Compute · CoreWeave GPU Pricing Review (jun. 2026) — https://www.thundercompute.com/blog/coreweave-gpu-pricing-review
Thunder Compute · NVIDIA H100 Pricing (Jun 2026) — https://www.thundercompute.com/blog/nvidia-h100-pricing
Spheron Blog · Lambda Cloud H100 Pricing 2026 — https://www.spheron.network/blog/lambda-cloud-h100-pricing-2026/
Spheron Blog · AWS H100 Pricing 2026: P5 Instance Cost — https://www.spheron.network/blog/aws-h100-pricing-2026/
Spheron Blog · Google Cloud A3 H100 Pricing 2026 — https://www.spheron.network/blog/google-cloud-a3-h100-pricing/
Spheron Blog · Azure H100 Pricing 2026: ND H100 v5 — https://www.spheron.network/blog/azure-h100-pricing/
GMI Cloud · H200 GPU Provider Pricing 2026 (CoreWeave, Lambda, Nebius, GMI) — https://www.gmicloud.ai/en/blog/h200-gpu-provider-pricing
Spheron Blog · GPU Spot Instance Arbitrage 2026 — https://www.spheron.network/blog/gpu-spot-instance-arbitrage-2026/
Thunder Compute · Cloud GPU Spot Instance Availability and Interruption Rates — https://www.thundercompute.com/blog/cloud-gpu-spot-instance-availability
Synergy Research Group · Neocloud Market Forecast to Approach 400B USD by 2031 — https://www.srgresearch.com/articles/neocloud-market-forecast-to-approach-400b-by-2031-driven-by-surging-ai-infrastructure-demand
Signisys · The Neocloud Revolution: How 20 Billion USD in GPU Providers Reshapes Cloud Market — https://www.signisys.com/blog/the-neocloud-revolution-how-20-billion-in-gpu-focused-providers-are-reshaping-the-cloud-market/
SemiAnalysis · The Great GPU Shortage: Rental Capacity (neocloud taxonomy original) — https://newsletter.semianalysis.com/p/the-great-gpu-shortage-rental-capacity
Spheron Blog · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
IntuitionLabs · H100 Rental Prices Compared 1,49-6,98 USD/hr Across 15+ Cloud Providers (2026) — https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

Mon, 15 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The cost per token of on-premise inference is $ \text{€/GPU-hour} \div \text{throughput} $. Throughput is a direct function of the GPU’s useful occupancy. On a generic node of 4×H100 SXM with a reference cost of ~11 € per GPU-hour (depreciation plus energy plus infrastructure), the cost curve over occupancy looks like this:

Useful occupancy	Effective throughput (tok/s)	Cost per 1M tokens
20 %	~700	~43 €
40 %	~1,400	~21 €
70 %	~2,500	~12 €
85 % (practical ceiling)	~3,000	~10 €

Doubling occupancy from 20 % to 40 % halves the cost per token, without buying more iron. At 70 % the cost competes with European cloud providers (~2.2 €/GPU-hour on-demand at Scaleway). At 20 % on-prem is four times dearer than renting. The lever is not the model, nor the precision: it is how many paid-for GPU-hours produce serving tokens.

1 · The fundamental identity

The cost per million tokens (CPM) in your own inference is not a list price. It is:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{ef}} \times 3600 / 10^6} $$

where $ C_{\text{GPU}} $ is the cost per GPU-hour (€/h), $ N_{\text{GPU}} $ the number of GPUs assigned to the service and $ T_{\text{ef}} $ the effective throughput (tok/s). Making the dependence on occupancy explicit:

$$ T_{\text{ef}} = T_{\text{pico}} \times \rho $$

where $ \rho \in [0,1] $ is the useful occupancy rate (the fraction of time in which the GPU is processing real serving tokens). The resulting identity:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{pico}} \times \rho \times 3600 / 10^6} $$

The direct consequence: CPM is inversely proportional to $ \rho $. Doubling $ \rho $ halves CPM. The numerator (cost of the iron) does not change.

The posts coste-por-token-y-por-request and capacity-planning-inferencia-llm-on-premise cover how to calculate $ C_{\text{GPU}} $ and the reference peak throughput. This article deals with $ \rho $: how to measure it, what limits it and how to raise it.

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

The field DCGM_FI_DEV_GPU_UTIL (field ID 203) appears in nvidia-smi as “GPU-Util”. Its official definition in the DCGM documentation:

“GPU Utilization” — percentage of time during which one or more kernels were executing on the GPU.

The problem for LLM inference: the decode phase is memory-bound. The GPU runs a kernel reading weights from HBM token by token; therefore GPU_UTIL registers close to 100 % even though the tensor cores are at 15 % of their capacity. The field measures activity, not useful work.

The distinction is critical for FinOps: an operator who sees GPU_UTIL 98 % assumes “saturated GPU, no room for more load”. The reality may be “the tensor cores are at 20 % and the bottleneck is HBM”, which leaves room for continuous batching or additional bin-packing.

The metrics that measure real occupancy are those of the _FI_PROF_* subsystem, available in DCGM 3.x with the profiling module enabled:

Table of relevant DCGM fields

DCGM field	Field ID	What it measures	Unit	Operational note
`DCGM_FI_DEV_GPU_UTIL`	203	% time with ≥1 active kernel	%	Misleading in LLM decode
`DCGM_FI_PROF_SM_ACTIVE`	1002	Ratio of cycles with ≥1 active warp per SM	0–1	Compute activity, not occupancy
`DCGM_FI_PROF_SM_OCCUPANCY`	1003	Resident warps / theoretical maximum per SM	0–1	Real intra-SM parallelism
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	1004	% cycles with tensor cores (HMMA) active	0–1	The real compute efficiency metric
`DCGM_FI_PROF_DRAM_ACTIVE`	1005	% cycles with HBM transferring	0–1	Memory saturation
`DCGM_FI_PROF_PCIE_TX_BYTES`	—	TX bytes over PCIe	bytes/s	Useful in PCIe inference
`DCGM_FI_DEV_FB_USED`	252	HBM used	MiB	VRAM budget
`DCGM_FI_DEV_POWER_USAGE`	—	Real consumption	W	For real energy cost
`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`	—	Bitmap of throttle causes	bitmap	Detects silent degradation

The _PROF_* fields require the DCGM profiling module and the right driver permissions. They are documented exhaustively in the NVIDIA DCGM field ID reference and in the sibling post observabilidad-gpu-dcgm-llm.

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

Metric	Typical decode value	Interpretation
`DCGM_FI_DEV_GPU_UTIL`	95–99 %	A lie: there is an active kernel
`DCGM_FI_PROF_SM_ACTIVE`	0.45–0.65	SMs with warps 45–65 % of the time
`DCGM_FI_PROF_SM_OCCUPANCY`	0.30–0.55	Resident warps at 30–55 % of the maximum
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	0.10–0.25	Tensor cores active only 10–25 %
`DCGM_FI_PROF_DRAM_ACTIVE`	0.75–0.90	HBM busy 75–90 % of the time

Reading: decode is memory-bound. The model weights are loaded from HBM for each token; HBM is saturated but the tensor cores wait. All of the “utilisation” in nvidia-smi comes from memory reads, not from compute.

3 · MFU and HFU: occupancy expressed in FLOPs

The canonical compute efficiency metric is MFU (Model FLOPs Utilization), defined in the PaLM paper (Chowdhery et al., arXiv 2204.02311, section 4):

$$ \text{MFU} = \frac{T_{\text{obs}} \times C_{\text{modelo}}}{P_{\text{pico}}} $$

where $ T_{\text{obs}} $ is the observed throughput (tok/s) and $ P_{\text{pico}} $ is the theoretical peak performance of the hardware (FLOP/s).

where $ C_{\text{modelo}} $ is the number of FLOPs per token in a complete forward pass. For a dense transformer with $ P $ parameters, the usual approximation (forward plus backward = 6P FLOPs per token; forward only = 2P):

$$ C_{\text{modelo}} \approx 2P \quad \text{(inference)} $$ $$ C_{\text{modelo}} \approx 6P \quad \text{(training, forward + backward)} $$

HFU (Hardware FLOPs Utilization) measures the FLOPs actually executed in hardware, including activation recomputation (gradient checkpointing). In training with recompute:

$$ C_{\text{hardware}} \approx 8P \quad \text{(forward × 2 + backward × 4)} $$

therefore HFU > MFU when there is recompute; they are identical without it.

Typical MFU values

Regime	Hardware	Typical MFU	Limiting regime
Training (large batch, BF16)	H100 SXM	35–50 %	compute-bound
Training PaLM 540B	TPU v4	46.2 %	compute-bound
Prefill inference (large batch)	H100 SXM	25–45 %	compute-bound
Decode inference (bs=1)	H100 SXM	3–8 %	memory-bound
Decode inference (continuous batching, bs=32–64)	H100 SXM	15–30 %	memory-bound, attenuated

Decode with batch size 1 has a single-digit MFU because the hardware spends most of its time waiting for HBM to deliver weights. Raising the batch size (continuous batching) amortises the weight read across more simultaneous tokens and raises MFU.

The Roofline model

The roofline places each operation in the space (arithmetic intensity, throughput):

Decode with batch size 1 falls in the memory-bound zone to the left of the ridge point. Raising the batch (continuous batching) moves the point to the right and up, closer to the roofline.

4 · TCO sensitivity to occupancy

Assumptions for the example: a generic node with 4×H100 SXM 80 GB, total node cost ~44 €/hour (5-year depreciation of ~220,000 €, energy ~4×700 W at ~0.12 €/kWh, plus rack/colocation infrastructure; see coste-por-token-y-por-request for the detail of the identity). Reference peak throughput with Llama 70B FP8 plus continuous batching: ~3,500 tok/s aggregate.

$$ \text{CPM}(\rho) = \frac{44\ \text{€/h}}{3500 \times \rho \times 3600 / 10^6} = \frac{44 \times 10^6}{3500 \times \rho \times 3600} = \frac{3.49}{\rho}\ \text{€/1M tok} $$

Occupancy $ \rho $	Effective throughput (tok/s)	On-prem CPM	On-demand cloud CPM (~2.2 €/GPU-h)
20 %	700	~17.5 €	~6.3 €
40 %	1,400	~8.7 €	~6.3 €
60 %	2,100	~5.8 €	~6.3 €
70 %	2,450	~5.0 €	~6.3 €
85 %	2,975	~4.1 €	~6.3 €

On-prem / cloud crossover point: with this example hardware, the on-prem cost advantage over a comparable European cloud starts at $ \rho \approx 55\text{-}60\,\% $. Below that, idle makes on-prem dearer than renting. The cloud column is flat because cloud bills per hour used, not per the throughput you extract from it.

Example figures with generic hardware. The real numbers depend on the depreciation price, the local cost of energy, the model and the precision. The structure of the curve, CPM inversely proportional to $ \rho $, is universal.

5 · Idle metrics: where occupancy is lost

Before applying levers, you need to know which kind of idle dominates. Three categories:

Type of idle	Symptom in metrics	Usual cause
Scheduling idle	`DCGM_FI_DEV_POWER_USAGE` low, `SM_ACTIVE` < 0.05	No requests in the queue; GPU waiting for work
Batching idle	`SM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low, `DRAM_ACTIVE` low	Batch too small; prefill stall between requests
Memory-bound idle	`DRAM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low	Normal decode; HBM is the bottleneck; insufficient batch size

Scheduling idle is the most expensive and the most directly attacked with bin-packing. Batching idle is attacked with continuous batching. Memory-bound idle in decode does not disappear entirely (it is the physics of the transformer), but it is attenuated with a larger batch size.

The inference engine metrics that complement the diagnosis from the service side are documented in observabilidad-gpu-dcgm-llm and anatomia-metricas-dcgm-vllm-anomalias.

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Lever	Effect on $ \rho $	When it applies	Operational complexity
Continuous batching	High: removes idle between requests; raises decode MFU from ~5 % to ~20 %	Always in inference; enabled by default in vLLM	Low (engine parameter)
Scheduler bin-packing (kube-scheduler / Kueue)	High: concentrates loads on fewer nodes; frees whole nodes for shutdown or rebalancing	Clusters with load that varies over the day	Medium (scheduler policy)
MIG (Multi-Instance GPU)	Medium: fills GPUs with light loads that previously lived alone on a whole GPU	Heterogeneous loads: embeddings + reranker + guardrail + large model	High (hot repartitioning not available)
Time-slicing	Low to medium: raises occupancy in dev/bursts; no isolation	Consumer GPUs (RTX 5090/4090); low-risk multi-tenant dev	Low
MPS	Medium: concurrent execution of multiple small processes; reduces context-switch overhead	Many small concurrent kernels on a datacenter GPU; trust between loads	Medium
Chunked prefill	Medium: interleaves prefill and decode; reduces the TTFT spike and raises throughput	Loads with a mix of short and long prompts	Low (vLLM flag)
Quantisation (FP16→FP8→INT4)	Indirect: raises peak throughput, which lowers CPM for the same $ \rho $	Models with quantised kernel support (native Hopper FP8)	Medium
Autoscaling (KEDA)	Keeps $ \rho $ high by scaling replicas according to the queue	Variable and predictable load; cluster with spare capacity	Medium

The post compartir-gpu-time-slicing-mps-mig details the three sharing mechanisms (time-slicing, MPS, MIG) with VRAM budgets.

Bin-packing with Kueue

Kueue (sigs.k8s.io/kueue) is the Kubernetes-native queue manager for GPU jobs. Its model of cohorts and nominal quotas allows active bin-packing: jobs accumulate in a queue and are launched only when there is a node that can take them whole, instead of fragmenting the load across partially occupied nodes.

BestFit packing in the ClusterQueue is configured with:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: gpu-prod
spec:
 preemption:
 reclaimWithinCohort: Any
 withinClusterQueue: LowerPriority
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16

The priority-based preemption policy ensures that production jobs displace experimentation ones when there is scarcity, keeping occupancy on the production nodes high.

MIG as a bin-packing lever inside the GPU

MIG makes it possible to fill an H100 with light loads that would otherwise live alone on a whole GPU. A profile of 3×2g.20gb + 1×1g.10gb on an H100 can host simultaneously a 7B FP8 model (~14 GB of weights), two embedding services and an INT4 guardrail, all with hardware isolation. Without MIG, each of those loads would take a whole GPU with an individual $ \rho $ below 10 %.

The profiles available on the H100 80 GB (SXM5) according to NVIDIA’s MIG User Guide:

MIG profile	Compute slices	Memory	Max. instances
`1g.10gb`	1/7 SMs	10 GB	7
`1g.20gb`	1/7 SMs	20 GB	4
`2g.20gb`	2/7 SMs	20 GB	3
`3g.40gb`	3/7 SMs	40 GB	2
`4g.40gb`	4/7 SMs	40 GB	1
`7g.80gb`	7/7 SMs	80 GB	1 (whole GPU)

MIG is not available on consumer GPUs (RTX 5090, RTX 4090). Time-slicing is the only sharing option on that hardware.

7 · Continuous batching: the effect on MFU

Continuous batching (also called iteration-level scheduling or in-flight batching) is the mechanism with the greatest impact on occupancy in inference. The idea: instead of waiting for a complete batch to finish before launching the next, the engine evaluates the pipeline after each token and replaces completed sequences with new requests from the waiting queue.

Quantified effect in vLLM:

Without batching (bs=1, static): decode MFU ~3–8 %; GPU idle between requests.
With continuous batching (dynamic bs 16–64): decode MFU ~15–30 %; the GPU almost never waits.
On pure prefill loads with a large batch (bs=128+): prefill MFU 25–45 %; it approaches the compute roofline.

The vLLM parameter --max-num-seqs controls the maximum number of sequences in the concurrent batch. Raising it increases occupancy until HBM becomes the bottleneck (watch for DCGM_FI_PROF_DRAM_ACTIVE > 90 % sustained).

8 · Hardware and scale: what applies to what

Hardware	MIG	MPS	Time-slicing	Continuous batching	Note
H100 SXM / H200	Yes (7 inst.)	Yes	Yes	Yes	On-prem datacenter reference
A100 SXM/PCIe	Yes (7 inst.)	Yes	Yes	Yes	Previous generation; HBM2e
L40S / L40	No	Yes	Yes	Yes	Ada Lovelace; no MIG; good price/VRAM
RTX 5090	No	Yes (limited)	Yes	Yes	Consumer; no MIG; does not scale in production
RTX 4090	No	Yes (limited)	Yes	Yes	Consumer; 24 GB VRAM; no MIG

The RTX 5090 and RTX 4090 illustrate the case of hardware that does not scale for multi-tenant with isolation: they do not support MIG, VRAM is scarce for models above 7B with a wide KV-cache, and TDP (600 W / 450 W) is high relative to throughput. For production inference at scale, the 4×H100 SXM is the reference node of this series.

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

Starting point: Llama 70B FP8 on 4×H100 SXM, a load of 8 requests/s with 512 output tokens on average, without continuous batching (static bs = 8):

Observed throughput: ~900 tok/s (decode dominant).
$ \rho_{\text{efectiva}} \approx 900 / 3500 \approx 0.26 $.
CPM: $ 44 / (900 \times 3600 / 10^6) \approx 13.6\ \text{€} $.

Same load, enabling continuous batching with --max-num-seqs 64:

Observed throughput: ~2,300 tok/s (the dynamic batch fills the gaps).
$ \rho_{\text{efectiva}} \approx 2300 / 3500 \approx 0.66 $.
CPM: $ 44 / (2300 \times 3600 / 10^6) \approx 5.3\ \text{€} $.

CPM reduction: from 13.6 € to 5.3 €, –61 %, with no hardware change and without touching the model.

The vLLM parameters relevant for raising effective throughput:

--max-num-seqs 64 # maximum concurrent batch
--max-num-batched-tokens 16384 # total tokens per iteration
--enable-chunked-prefill # interleaves prefill and decode

Sources

NVIDIA — DCGM Field Identifiers reference (v3.1), lista completa de DCGM_FI_* con field IDs y definiciones. https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA — GPU Profiling Metrics (Run:ai / DCGM), definiciones de DCGM_FI_PROF_SM_ACTIVE (1002), SM_OCCUPANCY (1003), PIPE_TENSOR_ACTIVE (1004). https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics
Chowdhery et al. — PaLM: Scaling Language Modeling with Pathways (MFU definition, sección 4). arXiv 2204.02311. https://arxiv.org/abs/2204.02311
NVIDIA — Multi-Instance GPU (MIG) User Guide (perfiles H100 SXM5, particionado, aislamiento). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (catálogo completo H100 80 GB). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator 24.9.0). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.0/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview (kernel concurrency, SM allocation). https://docs.nvidia.com/deploy/mps/latest/index.html
Kueue — Overview (sigs.k8s.io/kueue, bin-packing, cohorts, quotas). https://kueue.sigs.k8s.io/docs/overview/
vLLM Blog — vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (continuous batching, chunked prefill). https://blog.vllm.ai/2024/09/05/perf-update.html
GMI Cloud — NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis. https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Saxena et al. — LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2402.16363. https://arxiv.org/abs/2402.16363

GPU chargeback and showback in multi-tenancy: how to split cluster cost between teams

Mon, 15 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

A cluster of 4×H100 SXM at ~1.40 €/GPU-hour (depreciated capex plus energy) shared between three teams with different occupancy produces a monthly chargeback report of 3 rows and an additional idle row that nobody claimed. Without tooling, that idle cost vanishes, diluted into the total. With OpenCost + LiteLLM + Kueue, attribution operates on three orthogonal planes: the iron (OpenCost, €/GPU-hour per namespace/label), token consumption (LiteLLM, €/token per key/team/model) and the scheduler quota (Kueue, GPUs reserved per ClusterQueue). Crossing the three produces the number that goes to finance: team B consumed X million tokens at Y €/1M tok, the GPU cost it Z €, and it still has 2 GPUs borrowed from the cohort that will cost it W € if it holds on to them next month.

Showback vs chargeback: the FinOps Foundation definition

The distinction is not one of technology but of accounting formality (FinOps Foundation — Invoicing & Chargeback, FinOps Foundation — Data Analysis and Showback):

Concept	Definition	Moves money	Requires
Showback	visibility of consumption and its cost per team; the report arrives, the budget does not change	No	metrics + attribution
Chargeback	the cost is transferred to the team’s or product’s P&L as real expenditure	Yes	showback + accounting policy + financial system

Two points from the framework worth pinning down:

Showback is a requirement of any FinOps practice; chargeback is optional and depends on the organisation’s accounting policy supporting transfers between cost centres.
Neither is “more mature” than the other. The narrative that chargeback is the “grown-up” version is false according to the framework itself. The natural sequence is: showback → trust in the data → chargeback if policy allows it.

When to use each

Situation	Recommended mode
First attribution cycle; teams do not trust the data yet	Showback
Teams with their own budget in a financial system	Chargeback
Shared cluster with no P&L separation per team	Showback with visible idle cost
Teams with a committed GPU availability SLA	Chargeback (real reservation)
Research/experimentation workloads with no formal budget	Showback + threshold alert

GPU cost attribution with OpenCost

Allocation API: attribution parameters

The OpenCost /allocation API is the piece that turns Prometheus metrics into a cost report per dimension (OpenCost — API):

Parameter	Values	Use in multi-tenancy
`window`	`today`, `7d`, `lastmonth`, RFC3339 range	window of the monthly report
`aggregate`	`namespace`, `label:LABEL`, `annotation:NAME`, `pod`, `controller`	attribution dimension
`includeIdle`	`true` / `false`	adds an `__idle__` row to the report
`shareIdle`	`true` / `false`	distributes idle among the non-idle allocations (proportional to non-idle cost)
`idleByNode`	`true` / `false`	computes idle per node instead of per cluster
`filter`	`namespace:"llm-prod"`, `label:equipo:"datos"`	restrict to one specific team

Query for a monthly report per team, with idle visible as a separate row:

curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d resolution=10m

With shareIdle=false (the default), OpenCost returns idle as a separate __idle__ entry, which makes visible how much was paid for unused capacity. With shareIdle=true, that idle is distributed among the tenants in proportion to their non-idle cost: each absorbs its share of the waste, which is the right behaviour for a chargeback that penalises whoever is responsible for the idle.

Labelling inference pods

For aggregate=label:equipo to work, the inference pods need the right label. Two methods:

Label in the pod spec (the most direct):

# vLLM Deployment for the datos team
metadata:
 labels:
 equipo: datos
 producto: rag-prod
 modelo: llama-70b
 entorno: prod

Annotation (when the label is already taken by another convention):

# aggregate as an annotation
aggregate=annotation:finops.io/equipo

OpenCost supports aggregate=label:KEY and aggregate=annotation:KEY with the same syntax; the choice between the two depends on the cluster’s labelling convention.

Shared costs and idle: the three modes

API mode	Behaviour	When
`includeIdle=false` (default)	Ignores idle; total cost looks lower than it is	Never in production
`includeIdle=true, shareIdle=false`	Idle in a separate `__idle__` row	Showback: visibility of waste
`includeIdle=true, shareIdle=true`	Idle distributed among tenants proportionally	Chargeback: the team pays its share of idle
`shareIdle=true, idleByNode=true`	Idle distributed per node (finer-grained if there are dedicated nodes)	Chargeback with heterogeneous nodes

For infrastructure with nodes dedicated per team, idleByNode=true is fairer: the idle of a node dedicated to team A is not transferred to team B.

LiteLLM as the attribution point for token consumption

Virtual keys and teams

LiteLLM materialises token chargeback with four mechanisms (LiteLLM — Virtual Keys, LiteLLM — Setting Team Budgets, LiteLLM — Spend Tracking):

Mechanism	What it does
Virtual key with `max_budget`	monthly budget per key; the key is blocked once it is exhausted
`budget_duration`	reset window: `30d`, `7d`, `24h`; the proxy runs a daily cron that resets according to the duration
`team_id`	groups keys; spend accumulates in `LiteLLM_TeamTable` per team
`tags`	tag each request; they allow budgets per tag (cost centre)
Spend logs	one row per request with `team_id`, `model`, `prompt_tokens`, `completion_tokens`, `response_cost`

Configuring teams with a monthly budget

# litellm-config.yaml — per-team budgets with the declared on-prem cost
model_list:
 - model_name: llama-3-70b-onprem
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-svc:8000/v1
 input_cost_per_token: 0.00000140 # 1.40 €/1M tok (€/GPU-hour ÷ throughput)
 output_cost_per_token: 0.00000140
 - model_name: llama-3-8b-onprem
 litellm_params:
 model: openai/llama-3-8b
 api_base: http://vllm-8b-svc:8000/v1
 input_cost_per_token: 0.00000035 # 0.35 €/1M tok (1 GPU, higher throughput)
 output_cost_per_token: 0.00000035

litellm_settings:
 success_callback: ["langfuse"] # or any external logger

# Teams (via the /team/new API or config)
# POST /team/new
# {
# "team_alias": "equipo-datos",
# "max_budget": 800, # 800 € per month
# "budget_duration": "30d",
# "tpm_limit": 500000, # max tokens/min
# "rpm_limit": 1000 # max requests/min
# }

The input_cost_per_token key comes from the identity in article A4: the €/GPU-hour from OpenCost divided by the benchmark throughput. With that value, the response_cost of each request reflects the real cost of the on-prem iron for that model.

Endpoints for querying spend per team

# Team spend accumulated over the period
GET /team/info?team_id=equipo-datos

# Team spend per model (aggregated spend_logs table)
GET /global/spend/logs?team_id=equipo-datos&start_date=2026-06-01&end_date=2026-06-30

# Total spend across all teams
GET /global/spend/teams

The response includes spend (accumulated cost in the configured currency), total_tokens, prompt_tokens, completion_tokens and a per-model breakdown, the direct fields for the monthly chargeback report.

Joining €/GPU-hour (iron) with tokens per team

The two attribution planes, OpenCost (iron) and LiteLLM (tokens), are crossed with one key: the model_name and the namespace. The join happens outside the tools (in a BI pipeline or a SQL query over the LiteLLM database and the data exported from OpenCost):

coste_por_token_real = OpenCost.GPU_cost_namespace / LiteLLM.total_tokens_team

When the on-prem cost is properly declared in input_cost_per_token, LiteLLM already does this calculation internally and the response_cost is correct. The explicit join serves to validate: if the sum of LiteLLM’s response_cost per team does not match the GPU cost OpenCost assigns to the namespace, there is drift in input_cost_per_token that has to be corrected (changed throughput, a new optimisation, more replicas).

Kueue as the GPU budget/quota mechanism

ClusterQueue, LocalQueue and cohorts

Kueue introduces the quota-based scheduling layer on top of Kubernetes (Kueue — Cluster Queue, Kueue — Cohort, Kueue — Fair Sharing):

Object	Scope	What it does
`ResourceFlavor`	cluster	maps resources to a group of nodes (e.g. H100 nodes)
`ClusterQueue`	cluster	defines `nominalQuota`, `borrowingLimit`, `lendingLimit` per resource/flavor
`LocalQueue`	namespace	entry point for the team’s workloads; points to a `ClusterQueue`
`Cohort`	cluster	groups ClusterQueues that can lend quota to each other

Quota concepts

nominalQuota: GPUs guaranteed to the ClusterQueue at all times.
borrowingLimit: maximum additional GPUs it can borrow from the cohort when others are not using them.
lendingLimit: GPUs from its nominalQuota that it allows to be lent to others (if not specified, it can lend all unused ones).
Fair Sharing: a mechanism that orders pending workloads by the historical resource usage of their LocalQueue, giving preference to whoever has consumed less. Compatible with hierarchical cohorts since Kueue v0.11.

Example YAML: 3 teams in the llm-platform cohort

# ResourceFlavor mapping to the H100 SXM nodes
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
---
# ClusterQueue for the datos team — 2 guaranteed GPUs, can borrow up to 2
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-datos
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 queueingStrategy: BestEffortFIFO
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 2
 borrowingLimit: 2 # can use up to 4 GPUs in total
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# ClusterQueue for the ia team — 1 guaranteed GPU, lends what it does not use
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-ia
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 lendingLimit: 1 # lends its GPU when it is not using it
---
# ClusterQueue for the plataforma team — 1 guaranteed GPU, low priority
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-plataforma
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-plataforma
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 borrowingLimit: 3 # can use all 4 if the rest are free
---
# LocalQueue in each namespace — entry point for the workloads
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-datos

How Kueue materialises the “GPU budget”

FinOps concept	Kueue mechanism
Guaranteed budget	`nominalQuota`: the team always has these GPUs available
Maximum spend limit	`nominalQuota + borrowingLimit`: absolute ceiling of admissible GPUs
Lending idle capacity	cohort + `lendingLimit`: others take what this one does not use
Fair sharing between teams	`Fair Sharing` + `WorkloadPriorityClass`: those who have consumed most wait longest
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`: the owner of the lent GPU reclaims it by evicting lower-priority workloads

The nominalQuota is the expression of the budget in GPUs: if the team has 2 nominal GPUs and the price is 1.40 €/GPU-hour, its maximum guaranteed spend is

\[ \text{maximum nominal spend} = 2 \times 1.40 \times 720 = 2{,}016 \text{ €/month} \]

The borrowingLimit sets the potential overspend if it borrows from the cohort (and that borrowing can be charged with the same formula, multiplying the borrowed usage hours by 1.40 €/GPU-hour).

Cost model: an example with 4×H100 SXM and 3 teams

Node price

Generic hardware: a server with 4×H100 SXM 80 GB. 2026 market prices: a single H100 SXM costs between 27,000 and 40,000 € depending on the source (GMI Cloud — H100 GPU Pricing 2026); a 4×H100 server sits in the 120,000–180,000 € range including chassis, power supplies and NVLink. We use 140,000 € as a generic assumption for the complete server (4 GPUs plus infrastructure).

Component	Calculation	€/hour node
Depreciated capex (140,000 €, 4 years, 90 % availability)	140,000 ÷ (4 × 8,760 × 0.9)	~4.43
Energy (4 × 700 W TDP × PUE 1.4 × 0.12 €/kWh)	4 × 0.7 × 1.4 × 0.12	~0.47
Operations / networking / maintenance	estimate	~0.70
Total 4×H100 node		~5.60 €/h
Per GPU	5.60 ÷ 4	~1.40 €/GPU-hour

Cost-per-token formula for the 70B model (sustained throughput ~2,000 tok/s at TP=4):

$$\text{€/1M tokens} = \frac{5.60 \text{ €/h}}{2{,}000 \text{ tok/s} \times 3{,}600 \text{ s/h}} \times 10^6 \approx 0.78 \text{ €/1M tokens}$$

Monthly scenario: 3 teams, heterogeneous utilisation

Reference month (720 hours). The 4 GPUs are nominally assigned: 2 to Datos, 1 to IA, 1 to Plataforma. Real occupancy varies:

Team	Nominal GPUs	Average occupancy	GPU-hours used	Idle GPU-hours	GPU cost (€)
Datos	2	75 %	1,080	360	~1,512
IA	1	60 %	432	288	~605
Plataforma	1	35 %	252	468	~353
Cluster idle	—	—	—	1,116	~1,562
Node total	4	—	1,764	1,116	~4,032

Total node cost per month: 5.60 €/h × 720 h = 4,032 €. Cost per GPU-hour used: 5.60 ÷ 4 = 1.40 €.

With shareIdle=false (showback), the 1,562 € of idle appears as a separate row in the report and nobody pays for it directly. With shareIdle=true (proportional chargeback), it is split among the three in proportion to their non-idle cost: Datos absorbs ~857 €, IA ~333 €, Plataforma ~192 € of additional idle.

Token consumption per team (LiteLLM spend logs)

Team	Model	Tokens/month	Cost/token	Token cost (€)
Datos	llama-70b (TP=4)	380 M	0.78 €/1M	~296
IA	llama-8b (1 GPU)	210 M	0.35 €/1M	~74
Plataforma	llama-8b + batch	80 M	0.35 €/1M	~28

Monthly chargeback report (showback + tokens)

Team	Iron GPU cost (€)	Assigned idle cost (€)	LiteLLM token cost (€)	Total chargeable (€)
Datos	1,512	857	296	2,665
IA	605	333	74	1,012
Plataforma	353	192	28	573
Idle (without shareIdle)	1,562	—	—	visible, not charged

Attribution dimensions × policy table

Attribution dimension	Tool	Showback	Chargeback
namespace	OpenCost `aggregate=namespace`	`/allocation?aggregate=namespace&includeIdle=true`	`shareIdle=true` + transfer to P&L
pod label	OpenCost `aggregate=label:equipo`	report per label	same with shareIdle
virtual key / team	LiteLLM `/global/spend/teams`	token dashboard	hard `max_budget` per key
ClusterQueue (reserved GPU)	Kueue `nominalQuota`	observe usage vs quota	committed budget in GPUs
request tag	LiteLLM `tag_budgets`	per cost centre	budget per tag
pod annotation	OpenCost `aggregate=annotation:KEY`	report per annotation	same with shareIdle

End-to-end data flow

When the Datos team has exhausted its 2 nominal GPUs and borrows those of Plataforma (which is not using them), Kueue records that loan. As soon as Plataforma launches a new workload:

Kueue detects that cq-plataforma is below its nominalQuota.
With preemption.reclaimWithinCohort: LowerPriority, Kueue evicts the Datos workload that was on the borrowed GPU, if its priority is lower.
The evicted workload returns to the cq-datos queue and is readmitted once quota frees up.

Fair Sharing orders the pending queue by the LocalQueue’s accumulated historical usage: if Datos has consumed more GPU-hours than IA in the recent period, IA’s next workloads have admission priority over new ones from Datos. This implements an equitable split without blocking anyone permanently.

Configuring OpenCost for the node price

For the numbers in the previous section to be reproducible, the node price has to be declared explicitly (the default value underestimates on-prem GPU, issue #3781):

# OpenCost Helm values.yaml — on-prem 4×H100 node, 5.60 €/h
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 4xH100 SXM node"
 CPU: "0.025" # €/CPU-hour (minor component)
 RAM: "0.003" # €/GB-hour
 GPU: "1.40" # €/GPU-hour ← the one that drives the split
 storage: "0.0002" # €/GB-hour

Verify after configuring:

# Check the price resolved per node
curl http://localhost:9003/allNodePricing

# Attribution query per team label, last month, idle separate
curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d accumulate=true

Reference PromQL for chargeback dashboards

# GPU cost assigned per team (label:equipo) — €/hour
sum by (label_equipo) (
 container_gpu_allocation
 * on(node) group_left node_gpu_hourly_cost
)

# Idle GPU-hours per node (idle > 15 min)
sum by (node) (
 (1 - avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) / 100)
 * on(node) group_left node_gpu_hourly_cost
)

# Accumulated monthly cost per team (sum over window)
sum_over_time(
 sum by (label_equipo) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
 )[30d:1h]
)

Sources

FinOps Foundation — Invoicing & Chargeback Capability — https://www.finops.org/framework/capabilities/invoicing-chargeback/
FinOps Foundation — Data Analysis and Showback — https://www.finops.org/framework/previous-capabilities/analysis-showback/
FinOps Foundation — Allocation Capability — https://www.finops.org/framework/capabilities/allocation/
OpenCost — API (Allocation API, parámetros window/aggregate/shareIdle/idleByNode) — https://opencost.io/docs/integrations/api/
OpenCost — API Examples — https://opencost.io/docs/integrations/api-examples/
OpenCost — GitHub (issue #3781, infra-precio GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
LiteLLM — Virtual Keys (max_budget, budget_duration, team_id) — https://docs.litellm.ai/docs/proxy/virtual_keys
LiteLLM — Setting Team Budgets — https://docs.litellm.ai/docs/proxy/team_budgets
LiteLLM — Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM — Budgets & Rate Limits — https://docs.litellm.ai/docs/proxy/users
LiteLLM — Setting Tag Budgets — https://docs.litellm.ai/docs/proxy/tag_budgets
LiteLLM — Budget Reset Times — https://docs.litellm.ai/docs/proxy/budget_reset_and_tz
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Administer Cluster Quotas — https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/
GMI Cloud — NVIDIA H100 GPU Pricing 2026 — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
IntuitionLabs — NVIDIA AI GPU Prices H100 Cost Guide — https://intuitionlabs.ai/articles/nvidia-ai-gpu-pricing-guide

Sovereign on-premise vs hyperscalers: the case with data (cost, energy, performance and sovereignty)

Sun, 14 Jun 2026 05:30:00 +0200

Notation: amounts in euros (N €), decimals with a point; when a source quotes dollars it is marked “USD”. Data centred on Europe and Spain. The dollar symbol is not used (formula delimiter).

What this article covers

Synthesis article of the series (S2), and the heart of the proposal: the case with data for sovereign on-premise against the hyperscalers and the European cloud. Up to here, each track measured its own axis, cost per token (FinOps), goodput (benchmarking), energy and carbon; here the four are crossed, with a fourth dimension no US technical comparison puts up front: data sovereignty. The aim is to answer, with numbers and not with ideology, the question that underpins any investment in an AI platform: serve on your own iron, on a European cloud or on a hyperscaler? And to do it honestly: on-prem does not always win, and saying when it wins and when it does not is what makes the recommendation credible.

The framework: four axes, not one number

The mistake in almost every comparison is reducing the decision to the hourly cost of a GPU. The real decision crosses four axes, and only by seeing them together do you decide well:

Axis	Question	Who measures it in the series
Cost (TCO)	how much does it cost to serve, all in?	FinOps (A2–A8)
Performance	does it meet the SLO, at what goodput?	Benchmarking (B2–B8)
Energy and carbon	how many watts and grams per token?	Energy (C2–C8)
Sovereignty	under which jurisdiction does the data live?	GDPR / EU AI Act / CSRD

The first three are quantifiable and meet in the cost per token; the fourth is a constraint that can rule out an option however cheap it is. The synthesis consists of scoring each option on all four and deciding on the Pareto frontier, not on whichever axis happens to suit.

The cost axis: TCO and break-even

The real on-premise cost is amortised capex plus opex, and its cost per effective hour depends on utilisation:

$$\text{effective cost/GPU-hour (on-prem)} = \frac{\text{annual amortised capex} + \text{annual opex}}{8760 \times \text{utilisation}}$$

This formula is the key to the whole debate: the cost per useful hour of on-prem rises as utilisation falls, because capex is paid whether the GPU is working or idle. The 2026 data:

Data point	Value	Source
On-prem cost 8×H100 (floor, high util.)	~2.83 USD/GPU-hour all-in	Spheron
Annual on-prem cost (floor)	~237,000 USD/year	Spheron
AWS H100 (p5.48xlarge)	4.10–6.88 USD/GPU-hour	Spheron
AWS 8-GPU on-demand annual (100 % util)	287,000–482,000 USD/year	Spheron
Sovereign European cloud (Lyceum/Scaleway)	from 2–2.73 €/GPU-hour, zero-egress	Lyceum, Scaleway

The break-even against AWS on-demand falls around 50–83 % utilisation depending on region and tariff; below ~70 % utilisation, the cloud wins on TCO, and above it, on-prem (Spheron). For very high-utilisation workloads, on-prem pays for itself in less than 4 months (Lenovo).

Three-year TCO: the full calculation for an 8×H100 node

Abstract numbers do not convince a committee; a three-year model with declared line items does. Take a sovereign 8×H100 node in Spain and compare it, for the same work, with AWS and with a European cloud. Declared assumptions: amortisation over 3 years, energy at a solar PPA of 32.5 €/MWh (with grid backup), PUE 1.3, and two utilisation scenarios (50 % and 80 %).

On-premise (own node), annual line items:

Line item	Annual value	Note
Amortised capex (node ~270,000 € / 3 years)	~90,000 €	8×H100 server + network + storage
Energy (≈10.4 kW × PUE 1.3 × 8760 h)	~3,850 € (at 32.5 €/MWh)	with solar PPA; at grid tariff, ~12–18 k €
Operation, cooling, maintenance	~25,000 €	prorated staff, support, spares
Datacenter space (rack, connectivity)	~12,000 €	colocation or own datacenter
Annual total	~131,000 €	independent of utilisation

At a fixed 131,000 €/year, the cost per token depends only on how many tokens you generate, that is, on utilisation:

Utilisation	Useful GPU-hours/year	Cost/1M tokens
50 %	~35,000	~2.9 €
65 %	~45,500	~2.2 €
80 %	~56,000	~1.8 € (with cheap grid, ~1.1 €)

Sovereign European cloud (Scaleway/Lyceum), pay per use: at ~2.2 €/GPU-hour with zero-egress, the cost per token is constant with utilisation (you only pay for what you use): ~1.5–2.2 €/1M tokens depending on model and batching, with no capex and no idle risk.

Hyperscaler (AWS p5), on-demand: at 4.10–6.88 USD/GPU-hour (≈3.8–6.4 €), the cost per token works out at ~2–3.5 €/1M tokens, and on top of that you have to add the egress, quite apart from the fact that for GDPR data the sovereignty axis rules it out already.

The reading of the model is the thesis of the whole of S2: at 50 % utilisation, on-prem (~2.9 €) does not beat the European cloud (~1.8 €); at 80 % with cheap energy (~1.1 €), it beats it comfortably. The crossover sits, as the literature says, around 65–70 %. Investing in on-prem is, fundamentally, a bet that you will sustain high utilisation, and that bet is won with scheduling, not with hardware.

The uncomfortable reality: the utilisation almost nobody reaches

Here is the honest figure missing from the “on-prem is always cheaper” speeches: most inference teams in production operate at 40–65 % GPU utilisation, because of traffic variability and the limits of batching; the 80–90 % assumption that makes on-prem attractive is rarely reached outside batch-only pipelines (Spheron).

This changes the naive conclusion: if your real utilisation is 50 %, on-prem is not cheaper than the cloud, because the capex you pay for the idle GPU eats you. That is why utilisation is not a detail, it is the variable that decides the cost axis, and it connects directly with the FinOps track (the idle of A2, the chargeback of A5) and with scheduling: raising utilisation is what makes on-prem pay. A badly used cluster of your own is more expensive than the cloud; a well scheduled one is far cheaper. The cost question is not “on-prem or cloud?”, it is “can I sustain high utilisation?”.

The hidden costs of the cloud: egress

The cloud has its own small print: egress costs (taking data out of the provider). In the hyperscalers, moving data out or between regions is billed, and in AI workloads with a lot of data movement (datasets, checkpoints, embeddings) it can be a significant line item that does not appear in the GPU-hour price. The advantage of the sovereign European cloud: most of them (Lyceum among others) have adopted the zero-egress model, they do not charge for moving data out or between regions (Lyceum). When comparing, the real hyperscaler cost is GPU-hour + egress + other charges, not just the GPU-hour; ignoring it artificially inflates the hyperscaler’s competitiveness.

An example of the order of magnitude: a platform moving 50 TB/month outbound (datasets, checkpoints, responses served to systems outside the provider) at a typical egress tariff of ~0.08–0.09 €/GB pays ~4,000–4,500 €/month, that is ~50,000 €/year on egress alone, a line item the size of a third of the cost of a node of your own, invisible in the GPU-hour price. On the European cloud with zero-egress that line item is zero; on-prem, internal traffic is not billed either. That is why a fair comparison must model egress according to the real data pattern: for workloads with a lot of outbound movement, it can invert the ranking between hyperscaler and European cloud. The cloud bill is not the GPU-hour; it is the GPU-hour plus everything you move.

To this is added contract and lock-in risk: the hyperscaler’s on-demand GPU tariffs can change, commitment discounts (reserved/savings plans) tie you for 1–3 years, and migrating out, because of egress and coupling to proprietary services, has a real exit cost. On-prem and the European cloud with standard APIs (Kubernetes, S3 compatible) reduce that coupling: the same manifest and the same vLLM run on your cluster or on Scaleway without rewriting. Operational sovereignty, being able to move the workload without rebuilding it, is a value that does not appear in the tariff but weighs on a three-year decision.

The performance axis: the provider does not decide, goodput does

One point that simplifies the synthesis: performance does not depend on the provider, it depends on the hardware and the configuration. An H100 gives the same goodput on your cluster, on Scaleway or on AWS, served with the same vLLM and the same config. What decides performance is the goodput under your SLO (track B), not who hosts the GPU. So, in a comparison at equal hardware, the performance axis neutralises itself: what changes between options is cost, energy and sovereignty. The exception: if a provider gives you access to newer hardware (B200, GB200) ahead of your on-prem buying cycle, then the cloud can win on performance per GPU, a real argument in favour of the cloud for staying at the hardware frontier without capex.

The energy axis: the European and Spanish advantage

Here on-prem (or cloud) in Spain or France has a structural advantage over a hyperscaler in a dirty region. Recalling the data from the energy track:

Location	Grid carbon (gCO₂/kWh)	Price (indicative)
France (nuclear)	~20–60	low and stable
Spain (renewables + gas)	~150–170	low, volatile; solar PPA ~32.5 €/MWh
Germany	~363	high
Hyperscaler (region depends on provider)	depends; often not selectable	provider tariff

The same workload in France emits ~9× less carbon per token than in Germany, and in Spain, with a solar PPA at 32.5 €/MWh (an all-time low), the electricity cost, 30–50 % of the TCO, is low and, under contract, predictable. A sovereign cluster in Spain or France controls where the energy is consumed and with what carbon; a hyperscaler gives you the region it gives you, often with no choice of grid intensity. For CSRD reporting, that selectability is a quantifiable advantage of on-prem and the European cloud.

In concrete numbers: the example 8×H100 node (~10.4 kW × PUE 1.3 ≈ 118,000 kWh/year) emits, depending on the grid, ~2.4 t CO₂/year in France (~20 gCO₂/kWh) against ~43 t CO₂/year in Germany (~363 gCO₂/kWh), the same machine, the same work, ~18× the difference in reportable footprint just by choosing the location. That decision, which a hyperscaler in an imposed region does not let you take, is exactly what on-prem and the sovereign European cloud put in your hands. The energy axis is not an environmental detail: it is cost (the price of the kWh), compliance (CSRD) and sovereignty (control of location) all at once.

The sovereignty axis: the one that does not depend on utilisation

And here is the axis that invalidates the cheapest option if the data is sensitive. US hyperscalers are subject to the US CLOUD Act: US authorities can demand data held by a US company even if it sits in a European datacenter. For data subject to the GDPR, that is a compliance risk. Sovereign European clouds operate under EU/EFTA jurisdiction, providing data residency and GDPR compliance, and are exempt from the US CLOUD Act (Lyceum · sovereign providers). On-prem of your own is the maximum degree of sovereignty: the data does not leave your cluster.

The key difference from the other axes: sovereignty does not depend on utilisation or on volume. However much a hyperscaler cheapens the GPU-hour, for GDPR data it is not an option, the jurisdiction risk is not offset by price. It links with the ENS × ISO 42001 × EU AI Act controls and the EU AI Act mapping: compliance is a hard constraint, not an axis to optimise.

The four instruments that turn sovereignty into a concrete constraint rather than a slogan:

Instrument	What it requires	Implication for the architecture
US CLOUD Act	gives the US access to data held by US companies, wherever it is	a US hyperscaler does not guarantee jurisdictional residency even if the datacenter is in the EU
GDPR	residency and processing of personal data under EU law	requires an EU/EFTA provider or your own iron for personal data
EU AI Act	traceability, risk management and records for AI systems	favours the full control of the stack (logs, datasets, models) that on-prem gives
CSRD	verifiable reporting of environmental footprint	the selectable energy (clean grid, PPA) of on-prem and the European cloud is auditable

The operational conclusion: for a European entity processing personal data or deploying high-risk AI, three of the four technical axes can favour the hyperscaler and it can still lose, because the fourth axis, sovereignty, acts as a prior filter. That is why S2 orders the decision like this: first the sovereignty filter (which rules out the hyperscaler for GDPR data), then the optimisation of cost, performance and energy among the options that pass the filter (sovereign on-prem and European cloud).

The scorecard: the three options scored

Crossing the four axes for the three realistic options of a European platform (order-of-magnitude, illustrative figures):

Option	Cost/1M tok	Break-even	Energy/carbon	Sovereignty
Sovereign on-prem (ES/FR)	~1.1 € (high util.) / ~3 € (low)	>65–70 % util.	controllable (clean grid, PPA)	total (EU)
Sovereign European cloud	~1.5–2.2 €	no capex, pay per use	EU, zero-egress	high (EU)
Hyperscaler (US)	~2–3.5 € + egress	no capex	imposed region	not EU (CLOUD Act)

The reading of the scorecard: for data subject to the GDPR, the US hyperscaler is ruled out by the sovereignty axis, however competitive its tariff. The real decision comes down to sovereign on-prem vs sovereign European cloud, and there it is decided by utilisation and volume.

When each option wins

The honest recommendation, by scenario:

Scenario	Winning option	Why
High, sustained volume (util. >65–70 %), GDPR data	Sovereign on-prem	lowest cost/token + total sovereignty
Variable or growing volume, GDPR data	Sovereign European cloud	sovereignty without capex/idle risk
Low volume or sporadic peak	European cloud (per use)	you do not amortise the capex
No sovereignty requirement, hardware frontier	Hyperscaler	access to new hardware without capex
Hybrid (base + peak)	On-prem + European cloud (burst)	cheap base of your own, sovereign elastic peak

On-prem makes sense when there is very high, predictable utilisation (80 %+), strict sovereignty requirements, or a hyperscaler contract that works out expensive (Spheron). For sovereign platforms with a sustained base load, the winning pattern is usually the hybrid: sovereign on-prem for the high-utilisation base (where the cost per token is unbeatable) and sovereign European cloud for the peak and for growth (elastic, no capex, keeping EU jurisdiction). The best of both without giving up sovereignty.

Sizing the hybrid: how much on iron, how much on cloud

The hybrid is not “a bit of each”; it is sized with one figure: the base load percentile. The rule is to put on on-prem the load that is almost always present (the load that keeps the GPU at 75–85 %) and to send to the European cloud only the peaks that, if covered with iron, would leave GPUs idle most of the time. An example with a realistic traffic profile:

Load band	% of the time	Where to serve	Why
Base (p0–p70)	always	on-prem (1 node 8×H100 at ~80 %)	minimum cost/token, high util. guaranteed
Middle (p70–p95)	daily peak hours	on-prem if it fits, otherwise European cloud	elasticity without idle capex
Peak (p95–p100)	sporadic	sovereign European cloud (burst)	absurd to buy iron for a rare peak

With this split, the base amortises your own node at high utilisation (~1.1–1.8 €/1M tokens) and the peak is paid per use with no idle penalty (~1.5–2.2 €/1M tokens), all under EU jurisdiction. The expensive mistake is the opposite: sizing the on-prem for the peak, in which case the GPU spends most of its time idle at 30–40 %, the cost per token shoots up above 3 € and the cloud would have been cheaper. You size the iron for the base, not for the peak; the peak is exactly what the cloud does well. This principle connects with the capacity planning and the scheduling (Kueue/Volcano) of the series: the hybrid only works if the scheduler fills your own node before overflowing to the cloud.

Assumptions and sensitivity

The whole comparison hangs on assumptions that have to be declared, because moving them moves the conclusion:

Assumption	If it rises	Effect
Utilisation	50 % → 80 %	on-prem goes from losing to clearly winning
Energy price	expensive region → France/PPA	lowers on-prem TCO and carbon
Amortisation period	24 → 36 months	lowers the on-prem cost per hour
Volume	< 2M tok/day → much more	crosses the break-even towards on-prem
Egress (hyperscaler)	low → high	makes the hyperscaler dearer than the European cloud

The rule: no on-prem vs cloud comparison is valid without fixing these assumptions. One that says “on-prem is 3× cheaper” without declaring the assumed utilisation is propaganda; one that fixes utilisation, energy price, period and volume is a data point. The dossier must present the case with explicit assumptions and a sensitivity analysis, which is what makes it defensible before a committee that questions them.

Decision checklist

To take S2 from theory to decision, the questions that order the choice, in order:

Is the data subject to the GDPR, or is the system high-risk under the EU AI Act? If so, the US hyperscaler is ruled out on sovereignty; you choose between on-prem and European cloud. If not, the hyperscaler enters the cost comparison.
Can I sustain utilisation above 65–70 % on the base load? If so, on-prem wins on cost for that base. If not, the European cloud avoids paying capex for idle GPUs.
Does the traffic profile have marked peaks? If so, hybrid: base on iron, peak on European cloud. Size the iron for the base, never for the peak.
How much data do I take out of the provider each month? Model the egress; with a lot of movement, the zero-egress of the European cloud, or on-prem, win clearly.
Which electricity grid and at what price? France or Spain with a PPA lower TCO and carbon; include it in the model and in the CSRD report.
Have I fixed utilisation, energy, period and volume in writing? Without those four declared assumptions, the number is not defensible.

Whoever answers these six questions with data, not with intuition, has the case built. The series' recommendation for a sovereign European platform with a sustained base load is stable: sovereign on-prem for the high-utilisation base plus sovereign European cloud for the peak, with the hyperscaler reserved only for workloads with no sovereignty requirement where frontier hardware is needed without capex.

Limits and traps (data-driven)

Unrealistic assumed utilisation. The 80–90 % that makes on-prem win rarely happens in production (40–65 % typical). Model your real utilisation, not the ideal one.
Comparing only the GPU-hour. TCO includes energy, operation, cooling, egress (cloud) and capex (on-prem). Compare totals with the same assumptions.
Ignoring sovereignty. For GDPR data, the sovereignty axis rules out the hyperscaler before cost does; it is not negotiable with price.
Forgetting the hybrid. It is not “all on-prem or all cloud”; the base-plus-peak pattern usually dominates.
Data in USD. US comparisons are in dollars and with dirty regions; convert them to euros and to your region’s grid (Spain/France) for your case.

The synthesis of S2, in one sentence: for sovereign European data, the decision is not on-prem vs cloud in the abstract, but sovereign on-prem (high utilisation) plus sovereign European cloud (peak) against a hyperscaler that the sovereignty axis rules out, and utilisation is the variable that splits the base between the first two. The rest of the series gives the numbers for each axis; this one crosses them into the recommendation. The next synthesis article (S3) sizes the investment; this one decides the architecture.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Lenovo Press · On-Premise vs Cloud: Generative AI TCO (2026) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Lyceum · EU Sovereign Inference Platform Comparison (2026) — https://lyceum.technology/magazine/eu-sovereign-inference-platform-comparison/
Lyceum · Sovereign Cloud Providers 2026 — https://lyceum.technology/magazine/sovereign-cloud-providers-2026/
Scaleway · H100 GPU instance (precio €, soberanía UE) — https://www.scaleway.com/en/h100/
Nerd Level Tech · GPU Cloud TCO 2026: hidden fees, egress costs — https://nerdleveltech.com/gpu-cloud-comparison-2026-the-real-cost-of-ai-compute

The second cost vector of AI agents: durable execution with Temporal

Sun, 14 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

The thesis: agents have two cost vectors, not one

Almost all the cost discourse around agentic AI focuses on a single vector: the cost of inference (the tokens). And that one is solvable: FastMCP + vLLM on local hardware eliminates the cost of inference — you serve the models on your own iron, the cost per token drops to your amortised cost, with no API bill. But there is a second vector that almost nobody measures and that kills more projects: the cost of faulty execution. An agent is not a call, it is a sequence of steps (reason, call a tool, reason, call another one, and so on), and when that sequence fails halfway through with no persisted state, the cost is not linear: it doubles, it spirals with retries and it inflates on every recovery. This article is about that second vector, how durable execution attacks it (with Temporal as the reference), what the state of the art and the papers say, and why it is the real cause of so many cancellations.

The framing that backs the urgency comes from Gartner: it predicts that more than 40 % of agentic AI projects will be cancelled before the end of 2027, because of rising costs, unclear business value or inadequate risk controls (Gartner). “Rising costs” is exactly this second vector: not the cost of serving the model, but the cost of running it badly.

The three expenses of a faulty agent

The cost of faulty execution breaks down into three concrete sources, and it is worth seeing them with numbers:

Vector 1: failures charged twice

With no persisted state, a failure at step 15 of 20 costs you the tokens of the 14 previous steps (which you already paid for) plus the full retry from scratch. The wasted cost of an agent with no checkpoint, per failure, is:

$$\text{wasted cost} = \text{tokens of the completed steps} \times \text{cost per token} \times (\text{retries})$$

The longer the sequence and the later it fails, the more expensive: a failure at 75 % of the work throws away 75 % of the cost. Durable execution checkpoints every step: if the worker dies halfway through, another one picks the workflow up exactly where it stopped, not from scratch (Temporal). Temporal uses its Event History as the record of past decisions; on a failure it replays the progress to date and resumes at the exact point (CallSphere). Step 15 fails → step 15 is retried, not the previous 14. The wasted cost goes from “the 14 steps” to “the step that failed”.

Vector 2: uncontrolled retries

Agents that retry with no policy blow up the API or compute bill: a retry loop against a rate limit can generate dozens of billable calls for a single task. Durable execution defines retry policies per activity: hard limits on retries, exponential backoff and timeouts that bound the maximum possible cost of each call. Instead of “retry until it works” (unbounded cost), you define “at most N retries with backoff and a timeout of T” (bounded cost).

The state of the art backs this with data: the ReliabilityBench benchmark evaluates agent reliability under controlled failures and finds that rate limiting is the most damaging failure of all (arXiv 2601.06112). That is, exactly the failure that an unpolicied retry turns into a cost spiral. Bounding retries is not hygiene, it is the defence against the failure mode the literature singles out as the worst.

Vector 3: inflated context on recovery

With no persisted state, the agent rebuilds the full context on every recovery: it reloads the whole conversation/task history to “remember” where it was, paying the tokens of that enormous context on every recovery. With durable execution the state lives in the workflow history, so the next step receives only the context needed for that step, not the whole conversation. The saving is direct: fewer input tokens per step, and no penalty for rebuilding the context after each failure.

A worked example: the cost of one failure

Let us put numbers on the three vectors. A 20-step agent consuming ~3,000 tokens per step (reasoning plus tool call), on in-house inference at ~1.09 €/1M tokens, with a failure at step 15 and a failure rate of 10 % of executions:

Scenario	Tokens recomputed per failure	Cost per failure	Over 10,000 executions (10 % fail)
No state (full retry)	14 steps × 3,000 = 42,000	~0.046 €	~46 € wasted
With checkpoint (resumes at 15)	0 (only step 15 is retried)	~0.003 €	~3 € wasted

The absolute figure looks small because inference is in-house and cheap; but the ratio is ~15× less waste per failure, and it scales with the length of the sequence and with the cost per token (were it an external API at 10-30 €/1M, the waste without a checkpoint would be multiplied by 10-30). And this is only vector 1: add uncontrolled retries (vector 2), which can turn one failure into dozens of calls, and the context rebuilt on every recovery (vector 3). The cost of faulty execution is not the inference cost × something small; it is a multiplier that grows with the agent’s complexity, and it is exactly what does not show up in the demo and does show up in the production bill.

How Temporal works: workflows, activities and determinism

To understand why Temporal achieves this, its two primitives:

Workflow: the agent’s orchestration logic (the sequence of steps, the decisions). It has to be deterministic: given the same Event History, it produces the same decisions. That is what allows the execution to be replayed after a failure and to arrive at the same point.
Activity: every step with side effects (a call to the LLM, to a tool, to an API). Activities are not deterministic (the LLM is probabilistic), so their result is persisted in the Event History: on replay they are not executed again, they are read from the history.

The trick is that separation: the deterministic logic (workflow) is replayed cheaply from the history; the expensive, non-deterministic steps (activities, the model calls) are persisted and not repeated. That is why “completed work is never repeated”: completed activities are read, not recomputed. And retry policies are defined per activity, so each call to the model or to a tool has its own retry limit, backoff and timeout, which is the vector 2 cost control at the right level.

Durable execution: what it is and the landscape

Durable execution is a programming model that guarantees the code completes despite failures: every step is checkpointed automatically, and if a worker dies, another one picks the workflow up where it stopped (Temporal). It crossed into the “early majority” in 2025, with offerings from AWS, Cloudflare and Vercel, driven precisely by AI agent infrastructure (Inngest).

The tooling landscape (the 2026 reference architecture for durable LLM agents): Temporal, AWS Step Functions Express, Restate, DBOS and Inngest as durable execution primitives; and the checkpointer model of LangGraph (PostgresSaver, RedisSaver, DynamoDBSaver) on the agent framework side (render.com). Temporal stands out for its workflow-as-code model and its Event History as the source of truth.

Option	Model	Self-hosted	Note for sovereignty
Temporal	workflow-as-code, Event History	yes (open source)	the most mature; full on-prem
Restate	durable RPC/handlers	yes	lightweight, modern
DBOS	durable on top of Postgres	yes	state in your own Postgres
Inngest	event-driven, steps	SaaS / self-host	focus on DX
AWS Step Functions Express	state machine	no (AWS SaaS)	tied to AWS, not sovereign
LangGraph checkpointer	framework checkpointer	yes (your backend)	at framework level, not platform level

For a sovereign platform, the ones that are self-hosted (Temporal, Restate, DBOS) keep state in your cluster; the SaaS ones (Step Functions) move execution into someone else’s jurisdiction. Temporal is the choice for maturity and for its complete on-prem deployment.

The cost of durable execution itself

Honesty about the data: durable execution is not free. It has two costs to watch:

Execution overhead: persisting state and replaying history adds latency and compute. The reference architectures put it at around 5-20 % overhead on a load of 100,000 executions/day (render.com).
Price per step: in the offerings with per-step pricing, orchestrating many model calls, rate limit retries and complex interactions generates a large number of billable steps, and the cost can run away (render.com).

The conclusion for a sovereign platform: self-host the durable execution engine (Temporal is open source and deploys on-prem) and the SaaS per-step price disappears, leaving only the execution overhead, a 5-20 % that is paid back many times over by what it saves on the three vectors above. The arithmetic: if a failure halfway through a sequence used to cost you 50-75 % of the recomputed cost, a 5-20 % overhead is a bargain.

Observability: from manual post-mortem to real-time traceability

The fourth element of the equation: the observability layer. The integration with Braintrust and the complete history of every execution (Temporal’s Event History) turn cost analysis from a manual post-mortem, reconstructing by hand what happened and what it cost after a failure, into real-time traceability: every execution is recorded step by step, with its tokens, its retries and its cost, ready to analyse. This closes the FinOps loop of the cost-per-token article, but at the agent level: not just how much a token costs, but how much an execution costs, and where it is wasted.

The complete equation

Putting the pieces together, the cost architecture of a sovereign agent:

FastMCP + vLLM running locally eliminates the cost of inference. Temporal eliminates the cost of faulty execution. They are complementary, attacking different vectors of the same problem. Whoever solves only the first (serves cheaply but executes badly) keeps bleeding through the second; whoever solves only the second (executes robustly but pays for the API) keeps bleeding through the first. A sustainable platform needs both.

State of the art and papers

The reader asked for papers, and there are some: agent reliability and cost is an active research area in 2026.

Paper	Contribution
ReliabilityBench (2601.06112)	an agent reliability benchmark: consistency, robustness and tolerance to tool/API failures; rate limiting is the most damaging failure; ReAct more robust than Reflexion under stress
Cost-Efficient LLM Agents (arXiv)	the reliability-cost trade-off: routing every decision through the LLM improves accuracy but blows up cost; pre-coded graphs are cheaper but brittle against compound failures
Multi-agent Byzantine Fault Tolerance (2511.10400)	BFT consensus (CP-WBFT) to stabilise multi-agent systems under extreme failure rates (85.7 %)
ACRFence (2603.20625)	security of agent checkpoint-restore: preventing semantic rollback attacks
The Six Sigma Agent (2601.22290)	enterprise reliability via consensus-driven decomposed execution

Two readings for the proposal. First, the reliability-cost trade-off is real and is characterised: you cannot maximise accuracy ignoring cost, or the other way round; durable execution shifts the frontier by making reliability cheaper (checkpoint plus bounded retry instead of recomputing). Second, agent checkpoint-restore has its own security surface (ACRFence): persisting state introduces rollback risks that have to be controlled — durability is not only a cost decision, it is a security one too.

The sovereign angle

For a European sovereign platform the combination fits perfectly and is entirely on-prem: vLLM serves the models (no API cost and no sovereignty given away), FastMCP exposes the tools, and self-hosted Temporal orchestrates with durability (no SaaS per-step price and no execution data leaving for a foreign jurisdiction). The execution history, which describes in detail what your agent does and with which data, stays in your cluster, just like the inferences. Against a stack of agents on external APIs plus a US SaaS orchestrator, the sovereign stack removes both cost vectors and keeps the data under EU jurisdiction. It is the same argument as the rest of the series, applied to the agent layer: bounded, measured and sovereign cost.

FastMCP and the tool layer

The other component of the equation is FastMCP, the layer that exposes tools to the agent via MCP (Model Context Protocol). The agent does not call the APIs directly: it calls MCP servers that encapsulate each tool (a database, a search engine, an internal system) with a uniform contract. The connection with cost and durability: every call to an MCP tool is a Temporal activity, so it inherits its retry policy, its timeout and its persistence. That is, FastMCP defines which tools exist and Temporal governs how they are executed with durability and bounded cost. And because the MCP servers are your own, the tools and the data they touch stay on-prem, never exposed to an external provider. The triad vLLM (model) + FastMCP (tools) + Temporal (orchestration) is a fully sovereign agent stack: the model, the tools and the execution, all under your control and with the cost of both vectors bounded.

Patterns: human-in-the-loop and long sequences

Where durable execution shines most, by agent pattern:

Human-in-the-loop (HITL): human approval patterns, essential for an agent’s safety, map directly to the suspend/resume primitives of durable execution, allowing workflows that pause for hours or days waiting for an approval without losing state (Inngest). Without durability, keeping an agent “waiting” consumes resources or loses the context; with it, the workflow sleeps at no cost and wakes up with all its state.
Long sequences: the longer the chain of steps, the greater the checkpoint’s saving (a late failure throws away more work). Multi-step agents with tools are the canonical use case.
Multi-agent: when several agents collaborate, reliability gets complicated (the BFT papers study this); durable execution provides the consistent state substrate on which to build that coordination.

The pattern that does not need it: a single-step, stateless call (a classifier, a simple extraction). There the durability overhead buys nothing. The rule: durable execution for stateful sequences that fail halfway through; for everything else, a direct call.

How to measure the second vector

To manage the cost of faulty execution you have to measure it, and with the Event History plus observability (Braintrust) the KPIs are within reach:

KPI	What it indicates	Where it comes from
Tokens wasted per failure	the cost of vector 1	tokens recomputed after a failure
Retries per activity	the cost of vector 2	retry count from the Event History
Context size on recovery	the cost of vector 3	input tokens after a recovery
Failure rate per workflow	how often the problem occurs	failed executions / total
Cost per complete execution	the business figure	sum of tokens × cost/token per run

The last one closes the circle with cost per token: not just “how much a token costs”, but “how much an agent execution costs, and how much of that was useful work versus waste”. Without these KPIs the second vector is invisible, and what is invisible does not get optimised, it gets cancelled when the bill surprises you. With them, the cost of every execution is a real-time figure, and the decision to make each workflow durable (or not) is taken with numbers.

Limits and traps (data-driven)

Durability is not free. 5-20 % overhead; on SaaS with per-step pricing it can run away. Self-host Temporal to remove that price.
Not every agent needs it. A single-step, stateless task gains nothing from durable execution; the value is in the long sequences that fail halfway through.
Bounded retries can hide real failures. A retry limit avoids the cost spiral, but you have to alert when it is exhausted, not swallow the failure in silence.
Checkpoint = security surface. Persisting state opens the door to rollback attacks (ACRFence); protect it.
The first vector still exists. Temporal does not lower the inference cost; you need local vLLM for that. They are complementary, not substitutes.

The conclusion the data and the papers support: the cost of agentic AI has two vectors, and the industry has obsessed over the first (inference) while ignoring the second (faulty execution), which is precisely where Gartner locates the cause of 40 % of cancellations. FastMCP + vLLM solves the first; Temporal the second. A sovereign and sustainable agent platform needs to attack both, with observability that turns the cost of each execution into a real-time figure rather than an autopsy.

Closing

The dominant narrative about agent cost is half-lame: it treats the inference cost as if it were the only one, when it is only the one you see in the demo. The one that kills projects, the one Gartner files under “rising costs” inside its 40 % of cancellations, is the cost of faulty execution: failures charged twice, retries with no ceiling and the context rebuilt on every recovery. The engineering that attacks it is not a prompt trick or a better model, it is an execution model: durable execution, which checkpoints every step, bounds every retry and passes only the necessary context. Temporal, open source and self-hosted, does this while keeping state in your cluster, and combined with vLLM (model) and FastMCP (tools) it composes a fully sovereign agent stack where both cost vectors are bounded and the data never leaves the EU. The state of the art and the papers confirm the problem is real and characterisable: the reliability-cost trade-off, rate limiting as the worst failure, checkpoint-restore as a new security surface. For anyone who wants to be in the 60 % of agentic projects that are not cancelled, the recipe is the one in this article: solve both vectors, not one, and measure the second as seriously as the first.

Sources

Gartner · más del 40 % de proyectos de IA agéntica cancelados antes de 2027 — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Temporal · durable execution — https://temporal.io/
CallSphere · Temporal para workflows de agentes (Event History, replay) — https://callsphere.ai/blog/temporal-ai-agent-workflows-durable-execution-workflow-as-code
Inngest · durable execution, clave para agentes en producción — https://www.inngest.com/blog/durable-execution-key-to-harnessing-ai-agents
render.com · plataformas de workflow durables para agentes/LLM (overhead, precio por paso) — https://render.com/articles/durable-workflow-platforms-ai-agents-llm-workloads
ReliabilityBench (arXiv 2601.06112) — https://arxiv.org/abs/2601.06112
Cost-Efficient LLM Agents (arXiv 2603.01548) — https://arxiv.org/pdf/2603.01548
Byzantine Fault Tolerance multi-agente (arXiv 2511.10400) — https://arxiv.org/abs/2511.10400
ACRFence · checkpoint-restore de agentes (arXiv 2603.20625) — https://arxiv.org/pdf/2603.20625
The Six Sigma Agent (arXiv 2601.22290) — https://arxiv.org/html/2601.22290

From GPU-hour to cost per token: the metric that compares on-prem and cloud

Sun, 14 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Fourth article in the FinOps track (A4), and the one that closes out the metric that gives the whole pillar its meaning: cost per token. In A2 we saw how OpenCost assigns the cost of the iron (the pod, in €/hour); in A3, which tool to use. But the cost of the pod is not the metric that compares on-prem with cloud, nor the one the business understands: that one is cost per token (or per request). This article explains how it is really calculated (spoiler: for your own inference it is not a list price, it is €/GPU-hour divided by throughput), how to instrument it with a gateway (LiteLLM), how to attribute it by team and product, and how to compare it against European cloud. No recommendations; only the mechanics and the numbers.

The identity: from GPU-hour to token

Cost per million tokens (CPM) is the cost of the iron divided by what it produces:

$$\text{cost per token} = \frac{\text{cost of the iron (€/h)}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

This is the central formula of the FinOps track, and it has a consequence many people never quite internalise: for a self-hosted model, the cost of a token does not exist as a fixed price. It depends on two things that change: the cost of the iron (from OpenCost, in €/hour) and the throughput (from your benchmark, in tok/s). The same model on the same node costs half as much per token if you double the throughput with an optimisation, because the denominator changes, not the numerator. The cost per token of your own inference is a function of your efficiency, not a tariff.

That sets it radically apart from an external API, where the price per token really is a fixed tariff the provider charges you. Comparing the two requires turning your variable cost into a number, and that is where the iron (A2) meets the tokens (this article).

The two halves of cost per token

Cost per token comes from joining two measurements that live in different places:

The iron half comes from OpenCost (article A2): the by-usage cost of the vLLM pod, in €/hour. The token half comes from the gateway, which intercepts every request and counts tokens per model and per team. Divide, and out comes the cost per token and, crucially, per team, which is what enables chargeback.

The gateway: LiteLLM in depth

The gateway sits between the application and the engine (vLLM) or the provider, and intercepts every request. LiteLLM is the OSS reference: it proxies over 100 models and tracks spend by keys, users and teams, logging spend for all known models automatically (LiteLLM · Spend Tracking).

Its key mechanisms:

Mechanism	What it does
`response_cost`	the cost of each response, available in logging (`kwargs["response_cost"]`)
Spend logs	one row per request with tokens, model, user, key and cost (SQLite/PostgreSQL)
Tags	tag each request by team, project, environment and model tier
Budgets	budgets and rate limits per team or user
Custom callbacks	bespoke loggers for key/user/model/tokens/cost

With the spend logs table (via the database adapter) you have the detail of every request to build cost dashboards (LiteLLM · Logging), and you can tag each request with team/project/environment for per-team analysis, with notifications (Slack) and hard caps for keys that must not exceed a limit. In practice it is the platform’s token meter and cost allocator.

The self-hosted trap: LiteLLM does not know what your own token costs you

Here is the point almost nobody wires up correctly. LiteLLM tracks cost using list prices for known models (the API ones: OpenAI, Anthropic and so on). But for a model self-hosted on your vLLM there is no list price, because you pay nobody per token. If you tell it nothing, LiteLLM does not know the real cost of your own token. The solution is custom pricing: declaring to LiteLLM the cost per token of your self-hosted model (LiteLLM · Custom Pricing).

And where does that number come from? From the identity above: the cost of the iron (OpenCost) divided by the throughput (your benchmark). That is, the correct flow is:

OpenCost gives you the cost of the vLLM pod: ~11 €/hour (A2).
Your benchmark gives you the throughput: ~2,800 tok/s (track B).
The identity gives the cost per token: 11 ÷ (2,800 × 3,600 / 10⁶) ≈ 1.09 €/1M tok.
That 1.09 €/1M goes into LiteLLM as custom pricing.

Only then does the response_cost of each request reflect the real cost of your inference, rather than a zero or somebody else’s list price. Without this step, your LiteLLM cost dashboards lie about your own models. It is the explicit junction of the three tracks: iron (FinOps), throughput (benchmarking) and measurement (gateway).

Configuring custom pricing: an example

In LiteLLM, the real cost of a self-hosted model is declared in its configuration. An example with the cost per token derived from the identity (1.09 €/1M ÷ 10⁶ = 0.00000109 €/token):

# LiteLLM config.yaml — real cost of a self-hosted model
model_list:
 - model_name: llama-3.1-70b-onprem
 litellm_params:
 model: openai/llama-3.1-70b  # your vLLM endpoint
 api_base: http://vllm:8000/v1
 input_cost_per_token: 0.00000109 # €/token (from OpenCost ÷ throughput)
 output_cost_per_token: 0.00000109 # refine if you separate prefill/decode

With that, every response_cost LiteLLM records reflects the real cost of your iron. The operational note: this number must be updated when the cost of the node changes (new depreciation, a different energy price) or the throughput does (an engine optimisation). It is not a constant; it is the output of OpenCost and of the benchmark, and it is recalculated whenever either of the two changes. Automating that recalculation, from node cost and measured throughput to custom pricing, is what keeps cost dashboards honest.

Cost per token changes with throughput: read it off the sweep

Because throughput is the denominator, cost per token is not a point, it is a curve over the concurrency sweep. On the ~11 €/hour node:

Concurrency	Throughput (tok/s)	Cost per 1M tokens
1	350	~8.73 €
8	2,100	~1.46 €
16	3,330 (goodput)	~0.92 €
24	3,900 raw / 2,420 goodput	~0.78 € raw / ~1.26 € on goodput

Important readings: at low concurrency the cost per token is sky-high (the fixed cost of the node is shared among few tokens); cost falls as occupancy rises. But watch the last row: if you compute cost on raw throughput (3,900) you get 0.78 €, whereas if you compute it on goodput (2,420, the part that meets the SLO) you get 1.26 €, and the latter is the real cost, because requests that violate the SLO do not count as useful production. An honest cost per token is computed on goodput, not on catalogue throughput. This is where the cost track meets the benchmarking one: the correct denominator is goodput.

Cost per request: not only per token

Cost per token is the unit; cost per request is what a complete request bills. A request consumes input tokens (prompt) plus output tokens (generation), and both count:

$$\text{cost per request} = (\text{input tokens} + \text{output tokens}) \times \text{cost per token}$$

With nuances: in many engines the prefill (input) and the decode (output) have different costs per token (decode is memory-bound and more expensive per token), so a fine-grained cost model distinguishes the two. For most cases a weighted average cost per token will do; to optimise, it pays to separate prefill and decode. LiteLLM records input and output tokens separately in its spend logs, so the data is there to refine with.

Cost per request is what connects to the product: if a feature consumes 1,500 tokens on average (1,200 of prompt plus 300 of output) and the cost is 1.09 €/1M, each use costs ~0.0016 €, the basis for setting margin, as seen in the FinOps introduction.

Cost per model and per tier

A platform does not serve a single model. Cost per token varies by model and by configuration, and the gateway tells them apart. An 8B and a 70B on the same node have very different costs per token: the 8B yields far more throughput (more tokens/s) for the same GPU-hour, so its cost per token is a fraction of the 70B’s. The design consequence: routing each request to the cheapest model that meets the quality bar is a direct cost lever, since serving a 70B where an 8B suffices means overpaying per token.

Model (example, same node)	Relative throughput	Relative cost per token
8B	high	low
70B FP8	medium	medium
70B FP16	low	high

The gateway, with its per-model custom pricing, makes this difference visible and enables cost-based routing: classify the request and send it to the appropriate tier. It connects with the L7 router of the serving track: the gateway not only measures cost, it can decide on the basis of it. And quantisation (FP16→FP8) shows up again as a lever: the same model in FP8 raises throughput and lowers cost per token, moving the figure without changing model.

Attribution: from cost to team and product

The reason for putting a gateway in place is not only to measure total cost, it is to share it out. By tagging each request with team, product and environment, LiteLLM builds the split by business dimension:

With per-team budgets and hard caps per key, the gateway does not only report: it governs. A team that goes over budget gets an alert; a key with a hard cap cannot spend any more. It is the FinOps Operate phase applied to the token.

A combined report example

Joining the iron (OpenCost) and the tokens (gateway), the monthly cost-per-token-per-team report on the example node:

Team	Tokens/month	Applied cost per token	Charged cost
A · chat-prod	240M	1.09 €/1M (70B FP8)	~262 €
B · batch	90M	0.40 €/1M (8B)	~36 €
C · experimentation	15M	1.09 €/1M (70B)	~16 € + idle

What it reveals: A consumes a lot but at an efficient cost (FP8, high utilisation); B uses a cheap model (8B) for its workload; C consumes little but, as we saw in A2, drags along the idle cost of its underused GPU. The token report plus the OpenCost iron report give the full picture: who spends, on which model, with what efficiency. It is the chargeback that neither half would produce alone. And the per-team figure is what gets taken into the budget conversation: not “the cluster costs X”, but “your team consumed Y million tokens at Z €/million”.

Governing spend: budgets, caps and alerts

Measuring without governing is a dashboard; the value lies in closing the loop. LiteLLM allows three levels of control over spend:

Control	What it does	When
Soft budget	alert (Slack) when a threshold is crossed	tracking teams
Rate limit	limits requests/tokens per minute	avoiding cost spikes
Hard cap	the key cannot spend any more	keys that must never overrun

The healthy pattern: soft budgets with alerts for the teams (so they adjust their consumption), rate limits to smooth spikes, and hard caps only on critical keys (a service that must not run away). With this, cost per token stops being an after-the-fact figure and becomes a real-time control: the FinOps Operate phase, applied to inference.

On-prem vs European cloud comparison (the reason for all of it)

Cost per token exists to answer one question: is it cheaper to serve on my own iron or to rent? With the identity and European figures:

Option	Cost per 1M tokens	Notes
Depreciated on-prem (8×H100, 2,800 tok/s)	~1.09 €	your iron, your sovereignty
European cloud Scaleway (8×H100, on-demand)	~2.17 €	EU, no iron to operate
External API (proprietary model)	provider tariff	no data sovereignty

Cost per token is what makes the incomparable comparable: your variable cost (iron ÷ throughput) against a provider’s fixed tariff. And it exposes the lever: if an optimisation raises your throughput from 2,800 to 4,200 tok/s, your on-prem cost drops to ~0.73 €/1M, widening the advantage over renting. That is why benchmarking (track B) is, indirectly, the tool that moves cost per token the most: every goodput improvement translates into this number.

From cost per token to product price

Cost per token closes the loop with the business when it becomes cost per unit of product. The chain: cost per token (from the identity) × tokens per use of a feature = marginal cost of that feature. With that, three business decisions become data-backed:

Setting a price. If serving a feature costs 0.0016 € per use, the price to the customer has to cover it with margin; cost per token is the floor of pricing.
Spotting loss-making features. A feature that consumes many tokens (long prompts, lots of output) may cost more than it contributes; cost per request reveals it before you scale.
Comparing build vs buy at product level. On-prem cost per token against an API tariff, for that specific workload, decides whether to serve in house or consume externally.

This is the ultimate reason for the whole FinOps track: it is not infrastructure accounting, it is the quantitative basis for product and architecture decisions. A measured and attributed cost per token turns “is this profitable?” from an intuition into a calculation.

Detailed comparison: on-prem, European cloud and API

Extending the comparison with scenarios, for the same job (1M tokens):

Option	Cost/1M tok	Sovereignty	Where it fits
Depreciated on-prem, high utilisation	~0.7–1.1 €	full (EU)	high, sustained volume
Depreciated on-prem, low utilisation	~2–4 €	full (EU)	bad: idle drives cost up
European cloud (Scaleway, on-demand)	~2.2 €	EU	variable volume, no iron to operate
European cloud (reserved/committed)	~1.5–1.8 €	EU	medium committed volume
External proprietary API	provider tariff	non-EU	prototype / no sovereignty requirement

The table shows two things. First, on-prem only wins if utilisation is high, since at low occupancy idle makes it dearer than cloud, which connects cost per token with scheduling and utilisation (the idle of A2). Second, the external API, however competitive its tariff, loses data sovereignty, an axis that for GDPR data is not negotiable. Cost per token is the metric that puts the first three on the same scale; sovereignty is the constraint that rules out the last one for certain kinds of data.

Prompt caching and effective cost

One nuance refines the real cost per token: prefix caching. When many requests share the same prefix (a long system prompt, a repeated context), the engine caches its prefill compute and does not recompute it on every request. That means cached input tokens cost almost zero the second time and thereafter, so the effective cost per token of a workload with a high cache hit rate is lower than the nominal cost. External APIs already reflect this with a reduced price for cached tokens; in self-hosted the effect is that throughput rises (less prefill to recompute), which by the identity lowers cost per token.

The implication for the cost model: if your workload has shared prefixes (RAG over the same corpus, agents with the same system prompt), the real cost per token is lower than the naive calculation gives, and it pays to measure the prefix caching hit rate so as not to overestimate cost. And the reverse: a workload of unique, never-repeated prompts gains nothing from the cache, and its cost per token is the full nominal figure. Cost per token, once again, is not a constant: it depends on the model, the precision, the utilisation, the goodput and the reuse pattern of the workload. Measuring it on your real traffic, not on a synthetic benchmark without a cache, is what gives the number you actually pay.

State of the art 2026

The gateway as standard: LiteLLM (and alternatives) are consolidating as the layer that measures tokens, attributes cost and governs budgets in the LLM stack.
FOCUS for tokens: layers on top of the gateway generate FOCUS-compatible logs, integrating AI cost with the rest of cloud spend (FinOps introduction).
Custom pricing as a bridge: the OpenCost→gateway connection via custom pricing is the standard way to give self-hosted models a real cost.
Cost per token as a business KPI: increasingly, cost per token (and per product) is the metric the business demands, not the cost of the pod.

Limits and traps (data-driven)

Forgotten custom pricing. Without declaring to LiteLLM the real cost of your self-hosted token, the response_cost of your own models is zero or someone else’s price. Close the OpenCost→gateway bridge.
Static throughput. Cost per token changes with throughput; if you optimise the engine but do not update the custom pricing, the reported cost drifts out of date.
Forgetting prefill/decode. An average cost per token is fine for reporting; to optimise, separate input and output.
Measuring without governing. Spend logs without budgets or caps make a pretty dashboard; the value lies in closing the loop with alerts and caps.
Comparing tariffs with costs without matching assumptions. Your 1.09 €/1M assumes a utilisation and a throughput; the provider’s tariff does not. Fix the assumptions before comparing.

With cost per token solved, the FinOps track has its central metric; the following articles (A5 multi-tenant chargeback, A8 TCO model) build on it. The full chain, iron (OpenCost) ÷ throughput (benchmark) = cost per token, measured and attributed by the gateway, is what turns the cost of AI into a defensible number, in euros and per team.

Closing

Cost per token is the metric that reconciles the three tracks of the series into a single number, and its underlying lesson is counter-intuitive: for your own inference, the cost of a token is not a price, it is the reflection of your efficiency. It comes from dividing the cost of the iron (OpenCost) by useful throughput (the goodput of your benchmark), and it becomes visible and governable with a gateway (LiteLLM) that counts the tokens, attributes them by team and applies budgets. The mistake that invalidates the whole exercise is leaving custom pricing at zero and believing your own inference is “free”; the right move is to close the OpenCost→gateway bridge and keep the number up to date when the node cost or the throughput changes. Done that way, cost per token answers with data the one question the business keeps repeating, is it worth serving this on my own iron, and it does so in euros, per team and per product, comparable against European cloud and against any API. For a sovereign architecture proposal, it is the column of the dashboard that translates all the cost engineering into the sentence that closes a budget meeting: this is the cost per token, this is how it is measured, and here is the bench to reproduce it.

Sources

LiteLLM · Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM · Custom Pricing (modelos self-hosted) — https://docs.litellm.ai/docs/proxy/custom_pricing
LiteLLM · Logging — https://docs.litellm.ai/docs/proxy/logging
LiteLLM · documentación general — https://docs.litellm.ai/docs/
Statsig · LiteLLM cost tracking — https://www.statsig.com/perspectives/litellm-cost-tracking
StackPulsar · LiteLLM production monitoring (2026) — https://stackpulsar.com/blog/litellm-production-monitoring/

Kubecost vs OpenCost vs the alternatives: what the commercial layer adds and when it is worth paying for

Sun, 14 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article in the FinOps track (A3). A2 covered how OpenCost allocates cost, the free base. This article answers the question that comes next: is it worth paying for Kubecost or a commercial alternative, or is OpenCost enough? It is a build-vs-buy decision, operating the open source tool yourself against buying a product, and the answer depends on your scale, your audience and your appetite for operating infrastructure. No universal recommendations: here are the facts (what each one adds, what it costs, when it fits) and, for a European platform, the nuance of cost data sovereignty that the American comparisons do not mention.

The axis of the decision: build vs buy

OpenCost is free (Apache 2.0), but “free” means operating it yourself: deploying it, configuring the node price, keeping Prometheus with enough retention, building the Grafana panels and the alerts. Kubecost (and the alternatives) charge for taking that work away and adding capabilities OpenCost does not ship. The decision, therefore, is not “free vs expensive”: it is your own operating cost vs a licence.

Axis	Leans towards OpenCost	Leans towards commercial
Platform team	you have someone to operate it	you do not want to operate anything
Scale (cloud/GPU spend)	small to medium	large (the saving pays the licence)
Report audience	engineers	finance / management
Number of clusters	one	many (multi-cluster)
Need for automatic optimisation	you do it by hand	you want it out of the box

OpenCost is the open source core; Kubecost is the enterprise product built on that core, with proprietary features added; Kubecost was the original developer of the engine before releasing it, and IBM acquired the company (CloudZero · Kubecost vs OpenCost). In other words: both share the same allocation engine; what you pay for is the layer on top.

What Kubecost adds on top of OpenCost

On top of allocation (which is common to both), Kubecost adds a layer of optimisation, governance and enterprise features (CloudZero, Finout):

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓ (same engine)
GPU via DCGM	✓	✓ (3.0)
Invoice reconciliation (discounts, RI, spot)	—	✓
Rightsizing (recommendations)	—	✓
Anomaly detection	—	✓
Budget alerts	manual	✓
RBAC	—	✓
Multi-cluster aggregation	—	✓
Hosted option (SaaS)	—	✓
Long historical retention	your Prometheus	included
Support	community	commercial (IBM)

Invoice reconciliation

The capability that most justifies the price in cloud: OpenCost allocates on list rates; Kubecost reconciles against the real invoice, which includes discounts, reserved instances and spot prices (Finout). The difference can be large: a cost allocated at list rates can be 30–50 % above what you actually pay after discounts. On-prem this matters less (you declare the node’s real price), but in cloud or hybrid, reconciliation is the difference between a “theoretical” cost and the cost that shows up on the invoice.

Rightsizing and anomaly detection

Kubecost recommends resizing (rightsizing) underused resources, via the integration with IBM Turbonomic, and detects spend anomalies automatically. OpenCost gives you the data to do it by hand; Kubecost automates it. For a GPU fleet, rightsizing and anomalous idle detection are exactly where the saving of the Optimize phase lives.

The price of Kubecost (and the build-vs-buy economics)

Kubecost’s price starts at 449 USD/month for business use, with enterprise options on request (source). The right question is not “is 449 USD expensive?”, but “does operating OpenCost myself cost more or less than that?”. The build-vs-buy calculation:

Cost	OpenCost (build)	Kubecost (buy)
Licence	0	from ~449 USD/month (business)
Engineering time	deployment + maintenance + panels	minimal
Historical retention	your Prometheus/Thanos cost	included
Support	your team	included

If one platform person spends a few hours a month operating OpenCost, that time can already exceed 449 USD; if your cluster is small and the operating cost is nearly nil (you set it up once and it runs), OpenCost wins. The market rule of thumb: if monthly Kubernetes spend is below ~10,000 USD, the cloud-native tools or open source (OpenCost) usually suffice; above that, the saving the commercial tool captures pays for its licence ([search]).

Worked example: build vs buy

An illustrative calculation for a platform with a medium-sized GPU cluster:

Item	OpenCost (build)	Kubecost (buy)
Annual licence	0	~449 USD/month × 12 = ~5,400 USD (~5,000 €)
Initial deployment	~2–3 days of engineering	~half a day
Maintenance	~4–8 h/month (panels, alerts, upgrades)	minimal
Operating cost (at ~60 €/h)	~240–480 €/month	~marginal
Historical retention	cost of your own Thanos/Mimir	included

If OpenCost’s operating cost is around 240–480 €/month, it is in the same order as the Kubecost licence (~415 €/month). The decision, then, is not set by the price but by the added value: if invoice reconciliation, automatic rightsizing and the finance-facing UI save you or give you more than that difference, Kubecost wins; if your team already operates Prometheus/Grafana and your audience is engineers, OpenCost wins. For a large multi-cluster estate, the saving Kubecost captures usually far exceeds the licence; for a small and stable one, operating OpenCost is almost free. The number that decides is not 449 USD, it is your operating cost against that added value.

Migration and reversibility: an argument for starting with OpenCost

A rarely mentioned advantage of Kubecost being built on OpenCost: migrating between the two is low risk. Starting with OpenCost and, if scale justifies it, moving up to Kubecost later does not force you to redo the allocation, it is the same engine, the same concepts, the same Prometheus/DCGM instrumentation. What changes is the layer on top. This reduces lock-in: it is not an irreversible bet.

For a platform just starting out, the low-risk strategy is clear: stand up self-hosted OpenCost first, get the allocation right with the node price in euros, live with it for a few months, and only then decide whether invoice reconciliation, automatic rightsizing and the finance-facing UI justify paying for Kubecost. Starting with the commercial product “just in case” is paying before knowing whether you need it; starting with OpenCost and moving up if necessary is the option that keeps reversibility and discovers the real value before the invoice. And since both are self-hosted, neither route gives up data sovereignty.

The alternatives: CloudZero, Vantage, Finout

Kubecost is not the only commercial option; there is a category of cost platforms that operate at a different level (closer to the business than to Kubernetes):

Tool	Focus	Differentiator	Layer
CloudZero	unit economics	cost per feature/product/customer	business
Vantage	multi-cloud (20+ integrations)	breadth (incl. OpenAI invoice)	business
Finout	multi-cloud	virtual tagging (without touching resources)	business
Kubecost	Kubernetes	intra-cluster depth + optimisation	resource+optim
OpenCost	Kubernetes	open source allocation	resource

The key distinction: Kubecost/OpenCost are Kubernetes-centred tools (intra-cluster depth); CloudZero/Vantage/Finout are business cost platforms (multi-cloud breadth, cost per product). They do not always compete: many organisations use OpenCost/Kubecost for the Kubernetes detail and a business platform for the aggregate view.

A short profile of each:

CloudZero — a cost intelligence platform with a dedicated Kubernetes visibility solution for engineering teams. Its hallmark is unit economics: mapping cost to feature, product and customer, answering “how much does it cost me to serve this customer?” more than “how much does this pod cost?”. It fits when cost has to be taken to the product P&L.
Vantage — breadth of integrations (more than 20 sources: AWS, Azure, GCP, Kubernetes, Snowflake, Datadog and the invoice from LLM providers such as OpenAI). It is the option when AI cost is spread across many providers and you want a single view, including the invoice from the model APIs.
Finout — multi-cloud with virtual tagging: it applies cost labels without modifying the resources, which allows spend that was badly tagged at source to be allocated, and quick deployment. It is among the top Kubernetes cost optimisation tools (CloudBolt, Finout).

The usual pattern in a mature organisation: OpenCost/Kubecost for the Kubernetes and GPU detail, and a business platform on top to cross that cost with the rest of the cloud spend and take it to product. It is not “one or the other”: it is often “one and the other”, in layers.

Decision criteria

Five questions that settle the choice without opinion:

How much do you spend? Below ~10k USD/month, OpenCost is enough; above that, the saving pays for the commercial tool.
Who reads the report? Engineers → OpenCost; finance/management → Kubecost’s polished UI or a business platform.
One cluster or many? Multi-cluster leans towards Kubecost.
Do you want to operate infrastructure? If not, buy; if you have a platform team, build with OpenCost.
Where can the cost data live? The European question, below.

The European angle: cost data sovereignty

American comparisons leave out a criterion that for a sovereign platform is first-order: where the cost and usage data ends up. CloudZero, Vantage, Finout and Kubecost’s hosted option are SaaS, mostly American: you send them your telemetry of cost, usage, namespace names, teams and products. That data, which describes your operation in detail, leaves for a jurisdiction subject to the US CLOUD Act, with the same GDPR consideration as any other sensitive data sent to a hyperscaler.

Option	Where the cost data lives	Sovereignty
Self-hosted OpenCost	in your cluster	total
Self-hosted Kubecost	in your cluster	total
Hosted Kubecost (SaaS)	the provider’s SaaS	subject to the SaaS jurisdiction
CloudZero / Vantage / Finout	SaaS (US)	subject to the US CLOUD Act

It is worth being fair: cost data is not the same as user data. Spend telemetry per namespace is less sensitive than the content of the inferences. But it does reveal the operation: which teams exist, which products consume GPU, the size of the platform, and, crossed with namespace names, information that taken together can be sensitive. Most SaaS FinOps tooling is American; there are emerging European options, but few with the maturity of the big players. When in doubt, and for data subject to GDPR, the minimisation principle advises not taking out of the EU what does not need to leave, and cost allocation, with self-hosted OpenCost/Kubecost, does not need to leave.

The consequence for the proposal: self-hosted OpenCost or Kubecost keep the cost data under your control, just as the cluster keeps the inference data. If data sovereignty is a requirement (it is for GDPR data, see ENS × 42001 × EU AI Act controls), the American SaaS platforms run into the same conflict as a hyperscaler, however good their UI is. For Fibercli and any sovereign platform, this tilts the balance towards self-hosted, with OpenCost as the base and self-hosted Kubecost if the enterprise layer is wanted without giving up the data.

AI cost: what none of them ships out of the box

One point the generalist comparisons do not highlight and that is central for an LLM platform: none of these tools gives cost per token out of the box. OpenCost and Kubecost reach the pod cost (per resource/usage); CloudZero, Vantage and Finout reach the resource or invoice cost (including the OpenAI one, in Vantage’s case). But the jump from “this vLLM pod cost X €/hour” to “this request from this team cost Y” requires intercepting the inference traffic with a gateway (LiteLLM) that counts tokens per request and per team, as developed in the FinOps introduction.

Layer	What it gives	Who covers it
Resource/pod cost	€/hour per pod, namespace, team	OpenCost, Kubecost
Cloud/LLM invoice cost	€/month per provider	CloudZero, Vantage, Finout
Cost per token	€/1M tokens per team/model	gateway (LiteLLM) + any of the above

The implication for tool selection: cost per token, the metric that compares on-prem against cloud, is not solved by any tool on its own, but by the combination of allocation (OpenCost/Kubecost) with token measurement (gateway). When evaluating a commercial tool, the AI question is not “does it give the pod cost?” (they all do, one way or another), but “does it integrate well with my gateway to reach the token?”. Vantage, by ingesting the OpenAI invoice, gets close from the external API consumption side; for your own inference, the gateway remains essential.

State of the art 2026

Consolidation under IBM: Kubecost/OpenCost integrated into IBM’s FinOps Suite alongside Cloudability and Turbonomic; OpenCost remains the CNCF allocation standard.
First-class GPU: Kubecost 3.0 and OpenCost allocate GPU via DCGM; GPU FinOps stops being a special case.
Towards the token and FOCUS: the strong platforms track cost at token level, and the FOCUS standard (v1.3, Dec 2025) is extending to AI workloads, which will push interoperability between these tools.
A category of AI cost platforms: specific AI cost observability tooling is emerging that combines resource, invoice and token; it is worth demanding FOCUS compatibility so as not to couple to one provider.

Selection checklist

To decide without opinion, in order:

Step	Question	If…
1	Do I have a team to operate OpenCost?	no → commercial / hosted
2	Spend > ~10k USD/month and multi-cluster?	yes → Kubecost
3	Is the report for finance/management?	yes → Kubecost UI or business platform
4	Do I need cost per product/customer?	yes → CloudZero/Vantage/Finout on top
5	Can the cost data go to a US SaaS?	no → self-hosted (OpenCost / Kubecost)
6	Do I need cost per token?	yes → any of them + gateway

Step 5 is the filter for a sovereign platform: if the cost data cannot leave the EU jurisdiction, the list shrinks to the self-hosted options, with OpenCost as the base.

Pricing models of commercial tooling

For those that charge by saving or by percentage of spend:

Model	How it charges	Range
Fixed licence (Kubecost)	monthly rate	from ~449 USD/month (business)
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

The implication: at large spend, a 1–3 % fixed-fee can exceed Kubecost’s fixed licence; with a lot of waste, savings-based aligns incentives but can come out expensive if the saving is large. OpenCost, free, changes the equation if you have someone to operate it.

Coexistence: the reality is layered

In mature organisations, these tools are not chosen against each other, they are stacked. The usual pattern:

Layer	Tool	Answers
Intra-Kubernetes allocation	OpenCost / Kubecost	how much does each pod/team/GPU cost?
Per-token measurement	gateway (LiteLLM)	how much does each request/team cost?
Cost per product / multi-cloud	CloudZero / Vantage / Finout	how much does it cost to serve this product/customer?

Each layer solves a different question and feeds off the one below: the pod cost (OpenCost) plus the tokens (gateway) give the cost per token; that cost per token, aggregated by a business platform, gives the cost per product. Trying to make a single tool cover all three layers usually ends in compromises: the Kubernetes ones do not reach the business, the business ones do not reach the pod, and none reaches the token without the gateway. The healthy architecture is layers that integrate, not one tool that does everything by halves. And the glue between layers, in 2026, is FOCUS: demanding FOCUS compatibility from each piece is what allows the data to flow between them without being rewritten.

Summary by organisation profile

To get your bearings quickly, which stack fits each profile (indicative, not prescriptive):

Profile	Scale	Stack that usually fits
Small team with a platform	1 cluster, low spend	OpenCost self-hosted + Grafana
Scale-up with several teams	multi-team, medium spend	OpenCost + gateway; Kubecost if time is short
Multi-cluster enterprise	many clusters, high spend	Kubecost + business platform
Sovereign platform (Fibercli)	GDPR, data in the EU	self-hosted OpenCost/Kubecost + gateway; no US cost SaaS

The sovereign profile is the one that changes the answer relative to the standard guides: however much an American SaaS platform wins on features or on UI, the requirement to keep cost data under EU jurisdiction rules it out for GDPR data. For Fibercli, the bottom row is the one that rules, and that is why this series’ FinOps track is built on self-hosted tools.

Limits and traps (data-driven)

“Free” is not free. OpenCost has an operating cost; count it in the build-vs-buy or you are fooling yourself comparing 0 with 449 USD.
Reconciliation only matters in cloud. In pure on-prem, you declare the node’s real price and Kubecost’s reconciliation adds less.
Cost SaaS = cost data outside. For a sovereign platform, sending cost telemetry to an American SaaS reintroduces the jurisdiction problem that on-prem avoided.
Same engine, different UI. A good part of what you pay for in Kubecost is the presentation and governance layer; if your audience is engineers, it may not pay off.
Do not confuse layers. OpenCost/Kubecost (Kubernetes) and CloudZero/Vantage/Finout (business) solve different problems; sometimes they complement each other, they do not compete.

With the allocation tooling decision settled, the FinOps track moves on to cost per token (A4) and the full TCO model (A8). The foundation is the one from A2 and A3: correct allocation, with the node price in euros, in a tool that respects data sovereignty.

Closing

The OpenCost vs Kubecost vs commercial platform choice is sold as a feature comparison, and is in reality a build-vs-buy decision with a sovereignty constraint on top. The allocation engine is the same (OpenCost) in the first two; what you pay for in Kubecost is the optimisation, governance and presentation layer, which pays off when scale is large, the audience is financial or you do not want to operate infrastructure. The business platforms (CloudZero, Vantage, Finout) solve a different problem, cost per product, multi-cloud, and often coexist in layers with the Kubernetes ones rather than competing. But for a European sovereign platform there is a criterion no American comparison puts first and that for Fibercli comes first: where the cost data lives. Sending your spend, team and product telemetry to an American SaaS reintroduces, through the back door, the jurisdiction problem that on-prem avoided. The conclusion the data supports: self-hosted OpenCost as the base, self-hosted Kubecost if the enterprise layer is wanted, and the gateway to reach the token, a stack that keeps cost measured, in euros, and under EU jurisdiction. Expensive or cheap is secondary; sovereign or not is not.

Sources

CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
Finout · Kubecost vs OpenCost (6 diferencias) — https://www.finout.io/blog/kubecost-vs-opencost
Finout · Kubecost: pros/cons, pricing, alternativas (2026) — https://www.finout.io/blog/kubecost-pros/cons-pricing-tutorial-alternatives-2026-guide
CloudBolt · Top Kubecost Alternatives (2026) — https://www.cloudbolt.io/blog/top-kubecost-alternatives/
Amnic · OpenCost vs Kubecost — https://amnic.com/blogs/opencost-vs-kubecost
Clanker Cloud · Best tools for managing GPU usage in Kubernetes (2026) — https://clankercloud.ai/blog/best-tools-managing-gpu-usage-kubernetes-2025-2026-cost-roi

OpenCost in depth: how GPU cost is allocated in Kubernetes

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. European references (OVH is among the supported providers). The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Second article in the FinOps track (A2), and the first tool deep dive. The FinOps introduction established that OpenCost is the CNCF standard for cost allocation; here the box gets opened: how it allocates cost internally, where it gets its data from, how on-prem prices are configured in euros, what its API exposes, and where the traps are. Understanding the mechanics matters because it defines what any tool on top (Kubecost included) can and cannot do, and because a badly set base price invalidates the whole split. No recommendations; only the mechanics and the methodology.

What OpenCost is

OpenCost is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project). It is a cost allocation layer for Kubernetes: it reads resource usage from the cluster, joins it with a price, and splits the cost across Kubernetes dimensions down to the container (OpenCost · GitHub). It supports dynamic pricing via the billing APIs of AWS, Azure and GCP, and custom prices for on-prem clusters (OpenCost · configuration).

What it is not: it is not an optimiser, nor a governance system, nor a product unit economics tool. It is the piece that answers “how much does each namespace/pod/team cost?”, and leaves the rest to the layers above. That is why it is the foundation of the FinOps track: without correct allocation, there is no cost per token and no TCO model worth anything.

The cost model: node-level allocation

The key to OpenCost is that the model works at node level. It starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit per-resource prices, OpenCost uses the ratio of a set of base prices (marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem).

This is what makes it work on-prem: you declare the node cost (depreciated capex + opex), and OpenCost splits it without needing a cloud invoice. The normalisation guarantees that no cost is “lost” or “invented”: what you pay for the node is exactly what gets split across its resources.

Where the data comes from: Prometheus

OpenCost instruments nothing of its own: it reads from Prometheus, which is a prerequisite of the installation (OpenCost · GitHub). The sources:

Source	What it provides
`kube-state-metrics`	state of K8s objects (pods, requests/limits)
`node-exporter`	node resources and capacity
`cAdvisor`	real CPU/memory usage per container
`dcgm-exporter` (NVIDIA)	GPU usage, memory and power

With those series, OpenCost automates the join between Kubernetes resource usage and the provider price, reducing the custom PromQL you would otherwise write by hand (Grafana · manage costs). The GPU signal comes from DCGM via the dcgm-exporter (part of the NVIDIA GPU Operator), the same base as GPU observability. Without Prometheus and without DCGM exporting, OpenCost does not see the GPU.

Architecture: the pieces

OpenCost is not a monolith; it is three responsibilities worth telling apart:

Piece	Function
Cost model	resolves the node price and splits/normalises it across resources
Allocation API	serves the allocated cost per dimension (`/allocation` and endpoints)
Exporter	publishes cost metrics on `/metrics` for Prometheus/Grafana

The data flow: Prometheus holds the usage series (from kube-state-metrics, cAdvisor, dcgm-exporter); the cost model joins them with the resolved price; the Allocation API serves them aggregated by whatever dimension you ask for; and the exporter returns them to Prometheus so that Grafana can plot them. It is a cycle: metrics come in from Prometheus and cost goes back to Prometheus enriched. That symmetry is what lets you reuse your observability stack without standing up a new database: OpenCost lives on top of Prometheus, not beside it.

An operational note: historical analysis depends on Prometheus retention. For cost reports spanning months, a long-term backend behind it (Thanos, Mimir, VictoriaMetrics) is advisable, or periodically exporting the allocations to a store of your own.

On-prem prices in euros: the piece that decides everything

On-prem, the most important figure, and the most neglected, is the node price. It is configured per provider; the options are alibaba, aws, azure, gcp, oracle, ovh or default (on-prem) (OpenCost · configuration). For on-prem you use default, supplying base prices in a default.json (or by overriding the Helm values.yaml); anything you do not override uses the chart’s value (OpenCost · on-prem).

To set the node price in euros, the calculation is the TCO model one: depreciated capex + opex. An example of an 8×H100 node:

Component	Calculation	€/hour
Capex (node ~240,000 €, 36 months)	240,000 ÷ 26,280 h	~9.13
Energy (5.6 kW × PUE 1.4 × 0.058 €/kWh, France)	~0.455 €/h	~0.46
Maintenance, networking, operations	estimate	~1.5
Node total		~11.1 €/h

That ~11 €/h is the number you put into OpenCost, and everything is split from it. And here is the documented trap: with the default pricing configuration, OpenCost under-prices CPU and GPU on-prem (issue #3781). If you do not adjust the base prices to your real cost, the split will be systematically low and the cost per token you report will be unreal. Configuring the node price is not optional: it is the figure everything else hangs from.

Configuring the price: an example

The base price is given in the Helm values.yaml (or a default.json), in euros per hour and per unit of resource, derived from the node’s total cost:

# values.yaml — on-prem prices in €/hour (8xH100 node, France)
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 8xH100 node"
 CPU: "0.030" # €/CPU-hour
 RAM: "0.004" # €/GB-hour
 GPU: "1.30" # €/GPU-hour ← the figure that moves cost/token the most
 storage: "0.0002" # €/GB-hour

OpenCost normalises those values so that, multiplied by the node capacity, they add up to the total price you declared (~11 €/h in the example). The number to be most careful with is the GPU one: at 1.30 €/GPU-hour, the 8 cards are ~10.4 €/h, the bulk of the node’s cost. A badly set GPU base shifts the whole cost per token. Always review the resolved prices in /allNodePricing after configuring, because the default under-pricing bug bites exactly here.

Allocation: by usage, not by presence

The behaviour that has changed most and that matters most for GPU: in recent versions, the cost allocated to a GPU is determined by its usage, not its presence. If a GPU costing 100 €/month is only half used, the allocated cost is ~50 € ([search]). This aligns allocation with reality, you pay for what you use, but it has a consequence: the other half does not disappear, it is idle that somebody keeps paying for. OpenCost exposes it, and that is where the optimisation lever comes from.

OpenCost allocates to any Kubernetes dimension: cluster, node, namespace, controller (deployment, statefulset and so on), service, pod and container, and by labels (team, product). Access is through the Allocation API and several endpoints that expose the mechanics:

Endpoint	What it returns
`/allocation`	cost allocated by the dimension you ask for
`/costDataModel`	the resolved node price
`/allNodePricing`	hourly prices per node
`/pricingSourceSummary`	summary of the pricing source
`/metrics`	cost metrics for Prometheus

The /metrics endpoint is what turns OpenCost into a Prometheus exporter: once there, you can write PromQL to calculate the cost and efficiency of any Kubernetes concept and build panels in Grafana (OpenCost · exporter).

Allocation API parameters

/allocation is controlled by a handful of parameters worth knowing, because they define which number you get:

Parameter	What it controls	Example
`window`	time range	`7d`, `today`, specific dates
`aggregate`	aggregation dimension	`namespace`, `label:team`, `pod`
`accumulate`	sum the range or break it per interval	`true` (total) / `false` (series)
`idle`	include idle cost or not	`true` / `false` / `separate`
`filter`	filter by namespace, label and so on	`namespace:"llm-prod"`

The idle parameter is the most revealing: with idle=separate, OpenCost returns the idle cost as a separate row, which lets you see at a glance how much is being paid for unused capacity. A query with aggregate=label:team and idle=separate over window=30d is, literally, the monthly chargeback report with the waste highlighted.

GPU in depth: DCGM, usage, MIG and idle

The GPU is the expensive resource, so its allocation deserves detail:

Usage signal: dcgm-exporter gives DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (memory), and power. OpenCost uses utilisation to allocate by usage.
MIG (Multi-Instance GPU): an A100 is split into up to 7 isolated instances, which allows cutting idle from 50 % to almost 0 % by sharing one card across small workloads ([search]; see sharing the GPU: time-slicing, MPS and MIG). The cost allocation of each MIG instance follows the same usage logic.
Idle: the reliable detection pattern combines three pieces, DCGM metrics in Prometheus, an alert that fires when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, and routing of that alert to the team that owns the namespace ([search]).

An attribution nuance worth knowing: MIG creates isolated instances that Kubernetes sees as distinct resources, so OpenCost allocates them cleanly, each one to its pod. Time-slicing, by contrast, shares the same physical GPU across several pods with no compute isolation: there, per-pod utilisation is harder to separate, and cost attribution becomes approximate (the GPU is split across the pods sharing it, according to their measured usage). The practical rule: if you need exact per-team chargeback on a shared GPU, MIG gives cleaner attribution than time-slicing, at the price of rigid fixed partitions. It is a trade-off between cost accuracy and flexibility that is better decided before, not after, building the split.

Efficiency: the number that triggers action

With cost allocated by usage, OpenCost lets you calculate efficiency: how much of the allocated cost corresponds to useful work against the total. A GPU at 30 % utilisation has an efficiency of 30 %: the remaining 70 % is idle cost that somebody pays without receiving work. That figure, per team and per namespace, is what turns a dashboard into a chargeback conversation: “your namespace has 5 GPUs allocated at 25 % efficiency; either you raise utilisation or you release three cards”. It connects with FinOps and multi-tenancy: allocation locates the waste; scheduling and co-residency recover it.

Worked example: the allocation report

On the ~11 €/h node (8×H100), three teams share the cluster for a month. What a /allocation?aggregate=label:team&idle=separate&window=30d query returns:

Team	GPU-hours	Avg. util.	Allocated cost (€)	Efficiency
A · chat-prod	2,880 (4 GPU)	78 %	~3,744	high
B · batch	1,440 (2 GPU)	55 %	~1,872	medium
C · experimentation	1,440 (2 GPU)	22 %	~1,872	low
idle (separate)	—	—	~1,100	—

What the report reveals: the cost allocated to A, B and C comes out of their GPU-hours and the node price; but the idle row (~1,100 €/month) is capacity nobody used and everyone pays for in the pro rata. And the efficiency column points at C: 2 GPUs at 22 % is ~1,560 € a month of which only ~340 € is useful work. That is the actionable figure: C does not have a model problem, it has a utilisation problem. Without OpenCost, those ~1,100 € of idle and C’s inefficiency stay diluted in an aggregate invoice nobody questions. With it, they are a named row and a number in euros.

Useful PromQL queries

As a Prometheus exporter, OpenCost lets you build the panels by hand. Some reference queries (exact metric names vary by version; check in your /metrics):

# Cost per node and hour (resolved price)
node_total_hourly_cost

# GPU cost allocated per namespace (€/hour)
sum by (namespace) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
)

# Average GPU utilisation per namespace (for efficiency)
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

# Idle GPUs: utilisation < 10 % sustained
DCGM_FI_DEV_GPU_UTIL < 10

The last one, wrapped in an alert with for: 15m and routed to the namespace owner, is the standard idle detection pattern. With these series in Grafana you have cost per team, efficiency and waste on one dashboard, the “Inform report” of the FinOps phase in real time.

Dynamic cloud prices vs custom on-prem prices

The difference is worth understanding, because it changes how reliable the figure is:

Cloud (aws/azure/gcp/oracle/ovh): OpenCost obtains the price dynamically from the provider’s billing API, so the allocated cost reflects the real rate (including discounts, spot and so on) without you declaring anything.
On-prem (default/custom): you declare the price, derived from your TCO model. The advantage is total control; the risk is that a badly calculated price skews everything. That is why the TCO model (depreciated capex + opex) is not an academic exercise: it is literally OpenCost’s input.

The ovh provider among those supported is relevant for a European platform: if part of the workload goes to a European sovereign cloud, OpenCost can allocate its cost with dynamic pricing, and mix it with on-prem in the same allocation view.

From pod cost to cost per token

OpenCost gets as far as “this vLLM pod cost X €/hour”. The cost per token, the metric that compares on-prem against cloud, needs one more piece: the gateway (LiteLLM) that counts tokens per request and per team, as covered in the FinOps introduction. The full chain is: OpenCost gives the pod cost by usage → the gateway gives the tokens per team → dividing them gives the cost per token per team. OpenCost is the hardware half of that equation; without it, you would know the tokens but not their real cost. That is why A2 (allocation) is the foundation the whole cost per token of the proposal hangs from.

Deployment: Helm + Prometheus

The typical deployment is via Helm, with Prometheus as a prerequisite (for scraping and series storage). Conceptual steps:

Have Prometheus with kube-state-metrics, node-exporter, cAdvisor and dcgm-exporter.
Install OpenCost with Helm, pointing it at your Prometheus.
Configure the node price in euros (default.json / values.yaml), the step most often neglected.
Expose /metrics and build panels in Grafana with PromQL over the cost metrics.

As a Prometheus exporter, OpenCost lets you write PromQL for the cost and efficiency of any concept and create bespoke dashboards (OpenCost · exporter). It reuses infrastructure a GPU cluster already has for observability (DCGM, Prometheus, Grafana): OpenCost only adds the pricing and allocation layer.

OpenCost vs Kubecost (just the essentials)

OpenCost is the free base; Kubecost (IBM) is the commercial product built on it. Briefly, what the commercial one adds:

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓
GPU via DCGM	✓	✓ (3.0)
Long historical retention	depends on your Prometheus	included
Automatic rightsizing	—	✓ (Turbonomic)
Governance, alerts, enterprise RBAC	basic	✓
Commercial support	community	✓ (IBM)

For most on-prem clusters with a platform team, OpenCost covers allocation; Kubecost adds value when you want automated optimisation and enterprise features without operating it yourself. The detail of the comparison, and when paying is worth it, is article A3.

Implementation checklist

For OpenCost to give a defensible per-team cost, order matters:

Step	Action	Verification
1	Prometheus with kube-state-metrics, node-exporter, cAdvisor	series present
2	NVIDIA GPU Operator + dcgm-exporter	`DCGM_FI_DEV_GPU_UTIL` in Prometheus
3	Install OpenCost (Helm) pointing at Prometheus	`/allocation` responds
4	Configure the node price in € (capex+opex)	`/allNodePricing` shows the real one
5	Label pods/namespaces by team and product	`aggregate=label:team` splits correctly
6	Grafana panels for cost, efficiency and idle	the waste is visible
7	Idle alert (`util<10` `for: 15m`) to the owner	the alert reaches the team
8	(optional) long-retention backend	months of history available

Step 4 is the one most often skipped and the one that most invalidates the result. If you only do one thing right, make it that one: without the correct node price in euros, steps 5 to 7 split a wrong cost with great precision.

Limits and traps (data-driven)

Default under-pricing. With the default pricing config, OpenCost underestimates on-prem CPU and GPU (#3781). Adjust the base prices to your real cost or the whole split will come out low.
It depends on Prometheus and DCGM. Without the right series (especially dcgm-exporter), there is no GPU allocation. Input data quality rules.
Allocation by usage ≠ total cost. A GPU at 50 % charges half; the other half is idle that still costs. Do not confuse “allocated cost” with “cost paid”.
Prometheus retention. Historical cost analysis depends on how much your Prometheus retains; for long series, plan a long-term backend.
Allocation, not per-token measurement. OpenCost reaches the pod; cost per token needs the gateway (LiteLLM) above it, as covered in the FinOps introduction.

With OpenCost’s mechanics clear, the next article (A3) compares what Kubecost and the commercial alternatives add, to decide which FinOps stack to adopt. But the foundation, correct allocation with the node price properly set in euros, is this one.

Closing

OpenCost looks like “installing a cost tool” and is, in reality, declaring what your hardware is worth and letting it split itself. That simplicity is its strength, it lives on top of the Prometheus and DCGM you already have, with no new database, and also its trap: the split is only as good as the node price you declare, and the default value under-prices on-prem GPU. Properly configured in euros, OpenCost turns the cluster’s aggregate invoice into a split per team, per product and per GPU, separates used cost from idle, and exposes the efficiency that triggers chargeback conversations. It is the hardware half of the cost per token equation, the other half, the tokens, is supplied by the gateway, and without this half there is no TCO model or on-prem vs cloud comparison that holds up. For a sovereign architecture proposal, OpenCost is the piece that makes cost stop being an intuition and become a table in euros, reproducible and auditable, with the idle GPU named and identified. The Inform phase of GPU FinOps starts here, and everything else hangs on this number being right.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · configuración (proveedores: aws/azure/gcp/oracle/ovh/default) — https://opencost.io/docs/configuration/
OpenCost · configuración on-prem (precios personalizados) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exportador de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
OpenCost · issue #3781 (infra-precio CPU/GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Grafana · manage costs (OpenCost + Kubernetes) — https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

GPU/LLM FinOps: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The references are European (providers and prices from FR/DE/ES), as this is a sovereign proposal; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Second article in the data series, and the first pillar deep dive. Here the FinOps tooling for GPU/LLM infrastructure is inventoried with the detail an architecture decision needs: what metrics each tool handles, how it allocates cost internally, what GPU support it has, under what licence and pricing model it operates, and where its limits are. It is the natural continuation of the opening article, which established that cost per token is the unit that allows on-prem and cloud to be compared. No recommendations: the final choice is settled in the synthesis article with the Pareto table; here there are only facts and methodology.

Why GPU FinOps is a different problem

Classic cloud FinOps was born to split CPU, memory and storage, cheap and elastic resources. The GPU breaks three assumptions at once, and that is why it needs its own treatment:

It is expensive and discrete. An H100 costs on the order of 2.7 to 7 €/hour in European cloud (Scaleway from 2.73 €/h; OVHcloud somewhat more), or its depreciated capex on-prem; it is not a resource that gets split in trivial fractions. A 10 % allocation error across a GPU fleet is real money.
It is easily underused. Unlike the CPU, a GPU draws its power even when idle, and average occupancy in ungoverned clusters is notoriously low. The cost of idle, the GPU powered on without working, is waste number one, and it is invisible if it is not measured.
It is hard to attribute. Who owns the cost of a GPU shared by time-slicing or MIG across several pods from several teams? Without an allocation layer, GPU spend is a black hole nobody claims and nobody optimises.

The goal of GPU FinOps is to turn that black hole into an attributed, measured and actionable invoice: knowing what each team, each model and, in the end, each token costs.

The FinOps metrics

Metric	Definition	Unit
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	€ / 1M tok
Cost per request	cost charged to a complete request	€ / req
€/GPU-hour	hourly cost of a GPU (depreciated or rented)	€/h
Utilisation	fraction of the GPU actually used	% (MFU, GPU-hour util)
Cost of idle	GPU powered on without useful work	€/h wasted
Showback	reporting the cost to each team (without charging)	€ / team
Chargeback	charging the cost to each team	€ / team
Cost efficiency	real cost ÷ cost if it were at 100 %	%

The metric that closes the loop with the business is cost per token (or per request): it is the only one that can be compared across providers, across models and against the price of an external API. All GPU FinOps tooling exists to reach, one way or another, that number.

The three FinOps phases (FinOps Foundation)

The FinOps Foundation framework organises the work in a three-phase cycle. Applied to GPU:

Phase	Goal	Typical GPU action
Inform	visibility and allocation	measure cost per namespace/team/model/token
Optimize	reduce spend	rightsizing, spot, quotas, switch off the idle, quantise
Operate	continuous governance	budgets, idle alerts, automated chargeback

The common mistake is jumping to Optimize without having done Inform: GPUs get bought or replicas get adjusted without knowing where the money really goes. Allocation (the Inform phase) is the prerequisite for everything else, and it is where the tooling comes in.

How cost is allocated in Kubernetes: OpenCost’s mechanics

OpenCost is the de facto standard, so it is worth understanding how it allocates cost, because that defines what any tool on top can and cannot do. It is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project) (OpenCost · GitHub).

The model works at node level: it starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit CPU/GPU/RAM prices, OpenCost uses the ratio of a set of base prices (the provider’s marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem). This is key on-premise: you define the node cost (depreciated capex + opex) and OpenCost splits it.

It obtains utilisation by scraping Prometheus: kube-state-metrics, node-exporter and cAdvisor give it the real consumption per pod, and with that it allocates cost per cluster, node, namespace, controller, service or pod (OpenCost · exporter). For the GPU, the signal comes from DCGM (via the NVIDIA GPU Operator) exported to Prometheus, the same base as GPU observability. A common idle detection pattern: an alert when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, routed to the team that owns the namespace.

Frameworks, tool by tool

OpenCost — the allocation standard (CNCF, Apache 2.0)

What it measures: allocated cost of in-cluster resources (CPU, GPU, memory, volumes) by any Kubernetes dimension. Method: node-level pricing model + Prometheus scraping (above). GPU: yes, via DCGM. Licence: Apache 2.0, free. It can run as a metrics exporter to Prometheus with no further dependencies. Limit: it is the allocation layer, it brings no optimisation, governance or product unit economics; for that you put something on top of it.

Kubecost — the commercial product on OpenCost (IBM)

What it measures: what OpenCost does (it is built on it) plus enterprise capabilities. Kubecost 3.0 (2025) added GPU monitoring via NVIDIA DCGM and integration with IBM Turbonomic for automatic rightsizing, and extended its scope from Kubernetes to cloud service cost. IBM acquired Kubecost and integrated Kubecost/OpenCost into its FinOps Suite alongside Cloudability and Turbonomic (CloudZero · Kubecost vs OpenCost). Differentiator: rightsizing, governance, support. Limit: it is a commercial product; the bulk of the value over free OpenCost is the optimisation and enterprise layer.

CloudZero — unit economics and cost per product

What it measures: it maps cloud cost to features, products, teams and customers, not just to resources. Method: it ingests multi-cloud billing and models it in business dimensions. Differentiator: the deepest allocation and the unit economics approach (cost per business unit). Limit: less focused on intra-Kubernetes mechanics than OpenCost/Kubecost; it is the “cost to business” layer.

Vantage — multi-cloud with many integrations

What it measures: multi-cloud cost with more than 20 native integrations (AWS, Azure, GCP, Kubernetes, Snowflake, Datadog, OpenAI and so on). Differentiator: breadth of sources, including the invoice from LLM providers (OpenAI), which brings it closer to end-to-end AI cost. Limit: the depth of intra-cluster allocation is lower than that of K8s-native tools.

Finout — virtual tagging, fast deployment

What it measures: multi-cloud cost with virtual tagging: it applies cost labels without modifying the real resources, which allows spend that was not properly tagged at source to be allocated. Differentiator: fast deployment and flexible reallocation without touching the infra. Limit: like the others in its category, it depends on the quality of the billing data it ingests.

Comparison table

Tool	Scope	GPU	Licence / model	Differentiator	Layer
OpenCost	Kubernetes	Yes (DCGM)	Apache 2.0 (CNCF), free	allocation standard	resource
Kubecost	K8s + cloud	Yes (DCGM, 3.0)	commercial (IBM)	rightsizing, enterprise	resource+optim
CloudZero	multi-cloud	indirect	commercial	unit economics per product	business
Vantage	multi-cloud (20+)	via K8s/provider	commercial	breadth (incl. OpenAI)	business
Finout	multi-cloud	via K8s/provider	commercial	virtual tagging	business

Pricing models of commercial tooling

They are worth knowing because the cost of the FinOps tool is also FinOps:

Model	How it charges	Range
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

(CloudZero · FinOps Tools). The implication: at large cloud spend, a 1–3 % fixed-fee can exceed the savings-based model; at small spend with a lot of waste, savings-based aligns incentives. OpenCost, being free, changes the equation for whoever has a team to operate it.

FOCUS: the cost data standard

The cross-cutting problem of multi-source FinOps is that each provider bills in its own format. FOCUS (FinOps Open Cost and Usage Specification) is the open technical specification that defines requirements so that providers produce uniform billing datasets (FOCUS · FinOps Foundation). The committee ratified FOCUS v1.3 on 4 December 2025.

What is relevant for this series: at FinOps X 2026 the focus has been put on extending FOCUS to AI workloads, with token economics pushing the specification, and the expansion requests include AI, datacenter and SaaS/PaaS workloads (SiliconANGLE). In other words, the standard that normalises cloud cost is being stretched to cover AI cost per token. For an architecture proposal, betting on tools that emit and consume FOCUS is betting on future interoperability.

From resource to token: cost per token with a gateway

Resource allocation (OpenCost) gets as far as “this vLLM pod cost X €/hour”. To get to “this request from this team cost Y” you have to intercept the inference traffic. That is where the gateway comes in.

Tools such as LiteLLM sit between the application and the LLM provider/engine, and intercept every request to record tokens, latency and cost in real time. Layers above (for instance OpenLM’s AI Gateway) generate usage logs compatible with FOCUS (v1.0 to 1.3) and map spend to team, product, customer or feature, enabling showback or chargeback (OpenLM · token attribution). This links up with FinOps and multi-tenancy of the GPU cluster, where the gateway is the piece that splits cost between tenants.

Worked example: chargeback of a multi-tenant cluster

To see the two halves joined, a split over an example node (4×H100, depreciated cost 12 €/hour) shared by three teams via namespaces and MIG:

Team	GPU-hours allocated (OpenCost)	Hardware cost (€/h)	Tokens/day (gateway)	Cost/1M tok
A · chat product	50 %	6.00	8M	~0.75 €
B · nightly batch	30 %	3.60	3M	~1.20 €
C · experimentation	20 %	2.40	0.5M	~4.80 €

How it comes out: OpenCost splits the node’s 12 €/h according to the GPU-hours each namespace consumes (half for A → 6 €/h). The gateway supplies the tokens per team. Each one’s cost per million tokens is its hardware cost divided by its output, and it reveals something neither half would see alone: team C pays 6× more per token than A, not because its model is worse, but because its GPU is underused (a lot of allocated GPU-hours for few tokens). That 4.80 €/1M is the chargeback signal that triggers an optimisation conversation: either C raises its utilisation, or it releases the GPU. Without crossing allocation and tokens, that waste stays hidden in a cluster average.

The hidden cost: utilisation

Optimisation lever number one is not changing GPU, it is stopping paying for an idle GPU. As seen in the opening article, the electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %, and utilisation spreads all the fixed cost (capex + energy) over more tokens. Measuring idle is, therefore, the highest-return action of the Optimize phase:

Signal	Source	Typical threshold
Idle GPU	`DCGM_FI_DEV_GPU_UTIL`	< 10 % for > 15 min
Unused GPU memory	`DCGM_FI_DEV_FB_USED`	reserved but not used
Pods with no traffic	gateway metrics	0 requests, GPU allocated

Allocation (OpenCost) locates whose the idle GPU is; scheduling and co-residency (sharing the GPU: time-slicing, MPS and MIG) recover it. FinOps closes the loop: measure → attribute → optimise → govern.

Optimize: the saving levers (with data)

Once cost is allocated (Inform), the Optimize phase has a bounded repertoire of levers. Ordered by typical return:

Lever	Typical saving	Mechanism	Cost/risk
Recover idle	the largest	switch off/share the idle GPU	requires measuring utilisation
Quantisation (FP8/INT4)	raises throughput, lowers VRAM	more tokens per GPU-hour	possible quality cost
Reserved commitment	20–40 % over on-demand	reserve capacity	less elasticity
Spot/preemptible	the deepest discount	interruptible capacity	you have to tolerate cuts
Rightsizing	variable	match GPU type/count to the SLO	requires benchmarks
Autoscaling (HPA/KEDA)	variable	scale replicas with demand	tuning metrics

On commitment, the 2026 cloud data is conclusive: reserved plans give 20–40 % saving against on-demand, and spot the deepest discount in exchange for interruptibility; the price of the H100 fell 64–75 % between Q4 2024 and the start of 2026 (Spheron · GPU Cloud Pricing). Rightsizing and autoscaling connect with cluster scheduling: fitting the workload onto the right GPU and scaling replicas with demand are, at the same time, performance and cost levers. Quantisation appears here because it raises throughput, and, by the identity from the opening article, lowers cost and energy per token at the same time.

GPU FinOps maturity

A simple model to place where an organisation is, and what it is missing:

Level	State	What is missing to move up
0 · blind	aggregate invoice, no attribution	instrument OpenCost + DCGM
1 · visibility	cost per namespace/team	per-token measurement (gateway)
2 · unit economics	cost per token and per product	idle alerts and budgets
3 · optimisation	idle recovered, commitment, rightsizing	automated chargeback
4 · governance	chargeback + budgets + FOCUS	continuous improvement

Most organisations with GPUs are at level 0 or 1: they see a big invoice but do not know whose each euro is. The value jump is in reaching level 2, cost per token and per product, which is exactly where the tooling in this article stops being optional. The series’ FinOps track walks that ladder up to the full TCO model and the comparable cost/token that underpins the architecture proposal.

The minimum stack to reach level 2

Bringing the previous pieces together into a concrete toolchain, all open source, that takes you from blindness (level 0) to cost per token and per product (level 2):

Piece	Function	Alternative
NVIDIA GPU Operator + DCGM	exports GPU metrics (usage, memory, power)	—
Prometheus	stores the usage and cost series	VictoriaMetrics
OpenCost	allocates cost per resource and dimension	Kubecost (commercial)
Gateway (LiteLLM)	counts tokens per team/model	OpenLM AI Gateway
Grafana	cost, utilisation and idle panels	—

It is, deliberately, infrastructure many clusters already have for observability (DCGM, Prometheus, Grafana); GPU FinOps reuses that base and adds OpenCost and the gateway to it. It is not a new product, it is a layer on top of what exists.

KPIs to watch

KPI	What it indicates	Typical target
Average GPU utilisation	waste	>70–80 % sustained
Cost per 1M tokens (per model)	economic efficiency	compare against cloud rental
% of GPU-hours idle	money thrown away	minimise (<10–15 %)
Cost per team/product	attribution	fair split, no surprises
Deviation against budget	governance	alert before it is exceeded

These five KPIs are what turn GPU FinOps from a pretty dashboard into a decision tool: if you watch them, you know at all times what each thing costs and where the waste is.

State of the art 2026

Consolidation in Kubernetes: OpenCost is the CNCF allocation standard; Kubecost (IBM) the reference commercial product, already with GPU via DCGM and rightsizing through Turbonomic.
From resource to token: the strong platforms of 2026 track cost at token and GPU level, combining resource allocation (OpenCost) with gateway measurement (LiteLLM); it is the only route to a comparable cost/token for on-prem against cloud.
FOCUS v1.3 (Dec 2025) as the interoperability layer, extending to AI at FinOps X 2026: the cloud cost standard absorbs token economics.
Mature GPU FinOps: instrumentation leans on DCGM (the same base as observability), and idle detection by a utilisation threshold is common practice.

Cost per token: the bridge to the business decision

All the previous tooling exists to produce a number the business understands: cost per token (or per request). It is the one that lets you answer the three questions that underpin an architecture proposal:

Build or buy? On-prem cost/token (depreciated capex + opex, at real utilisation) against the price of an external API or of cloud rental. Below the volume threshold (~2M tokens/day) buying usually wins; above it, building.
How do you price a product? If a feature consumes N tokens per use and each million costs C, the marginal cost per use is N·C/10⁶, the basis of any margin.
Where is the waste? Cost/token per team (the chargeback example) points at whoever underuses the GPU before the aggregate invoice hides it.

The trap: comparing cost/token across scenarios without fixing the assumptions (utilisation, precision, ownership model). An on-prem cost/token calculated at 80 % utilisation is not comparable with one at 20 %, nor an FP16 with an FP8. That is why allocation (Inform) is not an end in itself: it is the raw material of a cost model with explicit assumptions, which is what the synthesis article turns into the “build vs buy” argument with defensible numbers. GPU FinOps is not accounting; it is the quantitative basis of the architecture decision.

Limits and traps (data-driven)

Allocation ≠ per-token measurement. OpenCost splits the resource cost; without a gateway you do not reach the request. They are two halves that have to be joined explicitly.
Badly set base prices = badly set allocation. On-prem, OpenCost splits the cost you declare for the node; if the depreciated capex/opex is wrong, the whole split is wrong. Input data quality rules.
Invisible idle. Without DCGM exported to Prometheus and without utilisation alerts, waste number one does not appear on any dashboard.
Cost of the tool. A 1–3 % fixed-fee on large spend is real money; compare the tooling cost with the saving it delivers (it is FinOps on top of FinOps).
Proprietary formats. Tools that do not emit/consume FOCUS tie you to their data model; in 2026 the robust bet is FOCUS interoperability.

The next article in the series (A2) goes into OpenCost in depth; this one fixes the map. With allocation settled, the rest of the FinOps track builds the TCO model and the cost/token the proposal needs.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · documentación on-prem (modelo de precio por nodo) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exporter de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
CloudZero · FinOps Tools: Definitive Guide (2026) — https://www.cloudzero.com/blog/finops-tools/
FOCUS · especificación (FinOps Foundation) — https://focus.finops.org/focus-specification/
SiliconANGLE · FOCUS y la economía de tokens de IA (FinOps X 2026) — https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/
OpenLM · atribución de tokens en tiempo real (LiteLLM + FOCUS) — https://www.openlm.com/enable-ai-finops-with-real-time-token-attribution/
Finout · Best AI Cost Observability Tools (2026) — https://www.finout.io/blog/best-ai-cost-observability-tools-in-2026

FinOps and multi-tenancy on the GPU cluster: who pays for what

Thu, 11 Jun 2026 12:20:00 +0000

Third instalment in an operational series on how to squeeze a generic on-premise LLM inference cluster of 4×H100 SXM 80 GB. The sibling pieces are Multimodal VLM on-premise with vLLM, serving vision-language models on the same cluster, and Speeding up cold start with Tensorizer, cutting the model load time from disk to HBM. This post closes the economic question the other two leave open: once the cluster serves several models for several teams, who pays for what?

TL;DR

An on-premise GPU cluster does not have the property that makes FinOps easy in the cloud: no monthly invoice arrives telling you what each compute hour cost. The cost has to be manufactured from capex and energy. The right unit is the token, and it is derived in two steps. First the €/GPU-hour: capex depreciation spread over the hours of useful life, plus energy (power × PUE × price_kWh). For an H100 SXM with an assumed capex of 30,000 € and 4 years of life at 90 % availability, that gives ≈ 1.07 €/GPU-hour (0.95 of depreciation + 0.12 of energy at 0.12 €/kWh and PUE 1.4). Second, the €/1M tokens: divide that €/GPU-hour by the sustained throughput. At 2,000 useful tok/s, ≈ 0.148 €/1M tokens; but that number only holds if the GPU is at 100 % useful utilisation. At 20 % utilisation the same token costs 4× more (0.74 €/1M), because capex and energy keep being paid even if the GPU is idle. That is the central thesis: the FinOps goal is not to negotiate the price of the token down, there is no provider to negotiate with, but to raise useful utilisation, which is the only term that moves cost by an order of magnitude. Attribution (chargeback / showback) is made real with LiteLLM virtual keys: per-team and per-user budget, RPM/TPM rate limit, automatic spend tracking and tags to assign spend to cost centres. Isolation between tenants is achieved with MIG (hard GPU partitioning), namespaces, ResourceQuota and PriorityClass. The FinOps Foundation rule worth carving in stone: showback always, chargeback only if the organisation’s accounting policy supports it; neither of the two is “more mature”. And the governance metric of the shared cluster is not the €/token but DCGM_FI_DEV_GPU_UTIL.

The problem the cloud hides from you and on-premise forces you to solve

In the cloud, the cost of a GPU is an observable fact: AWS, GCP or Azure charge you per GPU-hour and the invoice arrives itemised. FinOps in that world consists of reading an invoice that already exists and attributing it. On-premise there is no such invoice. The GPU was bought once, eighteen months ago, on a purchase order nobody remembers; the electricity bill arrives for the whole building with no breakdown of what fraction was the cluster’s; and the datacenter PUE is a number the facilities team knows but the AI team never asked about.

The first job of on-premise FinOps, then, is not to attribute a cost but to manufacture one. You have to build, from explicit assumptions, the equivalent of the “price per GPU-hour” that in the cloud is given to you. And once manufactured, you have to understand that this number is not a property of the hardware, like VRAM or TDP, but a function of how the hardware is used. An idle H100 costs exactly the same as an H100 at 100 %: the capex is already paid and idle energy is not zero. The only thing that changes is how many useful tokens that fixed cost produced. That is where the whole thesis of this post comes from.

The analogy: the coworking space with metered services

A coworking space manages a finite physical space and rents it to teams. The cost model has exactly the structure of the GPU cluster, and the analogy holds down to the last detail.

The desk-hour is the spread capex. The manager paid the rent on the premises, the furniture and the refurbishment, a one-off outlay or a fixed monthly cost, and spreads it over the available desk-hours in the month. Each desk has a base cost that does not depend on whether anybody uses it: the walls, the table and the chair cost the same empty as occupied. This is the depreciation of the GPU: capex divided by the hours of useful life.

The metered kWh is the energy. On top of the desk-hour, the coworking space measures the actual electricity consumption, the plugged-in laptop, the screen, the air conditioning in that room, and bills it separately. This is the energy of the GPU: power × hours × price_kWh, multiplied by the datacenter’s PUE, which is the coworking space also charging you for the fraction of the building’s air conditioning that cools your room.

The empty desk is paid for by whoever booked it. Here is the heart of the matter. If a team books ten desks for the whole month and only uses two, it pays for all ten anyway. The empty desk still takes up space the manager cannot resell, still depreciates furniture, still counts as committed capacity. In the cluster: a GPU assigned to a tenant that uses it at 20 % costs the same as if it used it at 100 %, and the idle 80 % is wasted capex that somebody is paying for.

The manager optimises occupancy, not tariff. A competent coworking manager does not obsess over lowering the price of the desk-hour, it is a sunk cost, the premises are already paid for. They obsess over the occupancy rate: every empty desk is margin lost forever, because an unsold desk-hour cannot be stored. The FinOps equivalent: the goal is not to lower the €/token, it is to raise utilisation, because the €/token falls only as a consequence of utilisation rising. There is no provider to squeeze; the only margin lies in not leaving GPUs idle.

The analogy has an honest limit worth naming: in a coworking space, two people cannot physically share the same chair. On a GPU you can, with time-slicing, MPS or MIG (see Sharing a GPU), and that is precisely the technical lever that lets you raise occupancy beyond what the physical coworking space would allow. But the economics of fixed cost + metered cost is identical.

The token as the unit of cost

Why the token and not the GPU-hour, or the request, or the model? Because the token is the only unit comparable across heterogeneous workloads. A GPU-hour of Llama 8B and one of Llama 70B produce radically different amounts of “useful work”; a RAG request with a 50-token answer and one generating a 3,000-token report are not comparable. The token, specifically the pair (input_tokens, output_tokens), normalises all of that. It is the unit LiteLLM accounts for natively, the one that appears in the OTel attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (see LLM tracing with OpenTelemetry GenAI), and the one commercial providers already use to charge. Adopting the token as the internal currency makes on-premise chargeback directly comparable with the cloud alternative, which is exactly the comparison management wants to see.

The two derived metrics that matter:

Tokens/€ — how many useful tokens each euro of total cluster cost buys. It rises when utilisation rises. It is the metric of economic efficiency.
Tokens/W — how many tokens each watt consumed produces. It is the metric of energy efficiency and, in a datacenter with limited power (the usual case on-premise), it is often the real hard constraint: you cannot add more GPUs because there are no more kW in the rack, so every watt has to perform.

Cost model of a GPU: from capex to €/token

The hourly cost of a GPU has two addends. Let us put all the assumptions in writing, because, as in capacity planning, a cost calculation without written assumptions is a disposable calculation.

Depreciation (spread capex)

$$\text{depreciation cost/h} = \frac{\text{capex}_{\text{GPU}}}{\text{hours of useful life}}$$

where the hours of useful life are the depreciation years × 8,760 h/year × the effective availability (the GPU is not available 100 % of the time: there is maintenance, restarts, update windows). Explicit, generic assumptions (they are not figures from any provider or any real purchase):

Capex per GPU: 30,000 € — a reasonable assumption that includes not just the GPU but its pro-rata share of the server, power supply, cooling, NVLink/InfiniBand network and rack. A “bare” H100 SXM costs less, but honest FinOps spreads the cost of the whole node across its GPUs.
Accounting useful life: 4 years. That is aggressive-realistic for a datacenter GPU; some depreciate over 3, others over 5.
Effective availability: 90 %.

$$\text{hours of life} = 4 \times 8{,}760 \times 0.90 = 31{,}536 \text{ h}$$ $$\text{depreciation cost/h} = \frac{30{,}000}{31{,}536} \approx 0.95 \text{ €/GPU-hour}$$

Energy (power × PUE × price_kWh)

$$\text{energy cost/h} = P_{\text{GPU}} \cdot \text{PUE} \cdot \text{price kWh}$$

Assumptions:

GPU power: the TDP of the H100 SXM is 700 W according to the NVIDIA datasheet (H100 datasheet). Under sustained inference load it hovers around that value; we use 0.7 kW as the average power under load. (At idle it drops a lot, 60–100 W, but the FinOps of reserved capacity reasons about power under load, which is what limits the rack.)
PUE (Power Usage Effectiveness): how many watts enter the datacenter for every watt that reaches the chip. A modest, well-managed on-premise datacenter sits at 1.4; a bad one at 2.0; the hyperscalers boast of 1.1. We use 1.4.
Price per kWh: 0.12 €/kWh — a generic assumption for an industrial tariff; it varies enormously by country and contract.

$$\text{energy cost/h} = 0.7 \cdot 1.4 \cdot 0.12 \approx 0.1176 \approx 0.12 \text{ €/GPU-hour}$$

The total €/GPU-hour

$$\boxed{\text{€/GPU-hour} = 0.95 + 0.12 \approx 1.07 \text{ €/GPU-hour}}$$

Two important readings of this number. First: depreciation dominates (89 % of the cost); energy is 11 %. This inverts many people’s intuition, who believe “the expensive part is the electricity”. With these assumptions, the expensive part is having bought the GPU, and that is why leaving it idle hurts so much. Second: the number is per-GPU; for the 4×H100 node it is ≈ 4.28 €/hour, and for the complete generic cluster of 4 nodes (16 GPUs), ≈ 17.1 €/hour ≈ 150,000 €/year of fixed cost that is paid whether it is used or not.

From €/GPU-hour to €/token

Now we divide the hourly cost by the tokens the GPU produces in an hour. Here sustained throughput comes in. Let us suppose, an order-of-magnitude figure, checkable with vllm bench serve, that a Llama 70B replica on 4×H100 (TP=4) sustains 2,000 useful tokens/s aggregated under good concurrency.

$$\text{tokens/hour} = 2{,}000 \times 3{,}600 = 7.2 \times 10^6 \text{ tok/h}$$

The cost of that replica (4 GPUs) is $4 \times 1.07 = 4.28$ €/hour. Therefore:

$$\text{€/1M tokens} = \frac{4.28}{7.2} \approx 0.59 \text{ €/1M tokens}$$

Per individual GPU, if we take a smaller replica (e.g. Llama 8B FP8 on 1 GPU at ~2,000 tok/s):

$$\text{€/1M tokens} = \frac{1.07 \text{ €/h}}{7.2 \times 10^6 \text{ tok/h}} \times 10^6 \approx 0.149 \text{ €/1M tokens}$$

These numbers are the theoretical floor at 100 % utilisation. Nobody operates at 100 %. And that is the whole story.

Utilisation is the lever (with numbers)

The €/token in the previous section assumes the GPU produces 7.2 M tokens every hour, hour after hour. In practice it produces that only during peak hours. The rest of the time it is partly idle: at night, at weekends, between traffic peaks. The useful utilisation $u$ is the fraction of the theoretical capacity that really turns into billable tokens.

The hourly cost is fixed (1.07 €/GPU-hour is paid come rain or shine), but the tokens produced scale with $u$:

$$\text{€/1M tokens}(u) = \frac{\text{€/GPU-hour}}{\text{max tok/hour} \cdot u} = \frac{1.07}{7.2 \times 10^6 \cdot u} \times 10^6 = \frac{0.149}{u}$$

The requested contrast, worked out:

Useful utilisation $u$	Real tokens/h	€/1M tokens	Multiplier vs 80 %
100 %	7.2 M	0.149 €	0.80×
80 %	5.76 M	0.186 €	1.00× (baseline)
50 %	3.6 M	0.298 €	1.60×
20 %	1.44 M	0.744 €	4.00×
10 %	0.72 M	1.488 €	8.00×

Direct reading: going from 20 % to 80 % utilisation divides the cost per token by four ($0.744 \to 0.186$). No tariff negotiation, no quantization, no hardware change gives you that factor of 4 so cheaply. Quantizing from BF16 to FP8 can double throughput, and therefore halves the €/token, but it degrades quality and demands evals (see Quantization for inference); raising utilisation from 20 to 80 % does not touch the quality of a single token.

This reformulates GPU cluster FinOps in a single sentence: the €/token is not something you negotiate, it is something you earn by filling the GPUs. And raising utilisation is an engineering problem that already has its pieces:

Autoscaling that switches replicas off when traffic drops, so as not to pay for idle GPU (see LLM autoscaling on Kubernetes with KEDA). Switching off a 4-GPU replica for eight hours overnight saves ≈ 34 € a day of real cost; multiplied by replicas and by days, that is the difference between a profitable cluster and one that bleeds.
Sharing a GPU between small workloads with MIG or time-slicing, so that two tenants that individually would use 30 % together fill a GPU to 60 % (see Sharing a GPU).
Overnight batch that fills the off-peak hours with non-urgent work (re-embeddings, evals, light fine-tuning) instead of leaving the GPUs switched off or ticking over.
Reducing cold start so that scaling to zero and starting again is cheap and therefore viable as a utilisation strategy (see the sibling piece Speeding up cold start with Tensorizer).

The diagram: the €/token cascade by utilisation

The cost per token is not a number, it is a cascade that multiplies as utilisation falls:

Attributing the cost: LiteLLM virtual keys

With the €/token manufactured, it remains to attribute it to whoever consumed it. The piece that already sits in front of the engines, the L7 inference router, is also the natural place to count. LiteLLM Proxy makes chargeback real with four mechanisms, all documented:

Virtual keys with a budget. Each team or user receives a virtual key with a max_budget and a budget_duration. You can define several independent budget windows (for example one of 24 h and another of 30 d) that reset on their own cycle (Virtual Keys). When a team exhausts its monthly token budget, the proxy rejects with a budget exceeded error instead of carrying on spending GPU-hours nobody will be able to charge anywhere.

RPM/TPM rate limit. Each key carries requests per minute (RPM) and tokens per minute (TPM) limits (Budgets, Rate Limits). This is not just anti-abuse protection: it is the tool that stops a noisy tenant monopolising the cluster’s utilisation at everybody else’s expense. The rate limit is the capacity quota; the budget is the spend quota. They are orthogonal and you use both.

Automatic spend tracking. LiteLLM accounts for the spend of all known models by key, user and team, recording the API key, the user, the team_id, the request’s tags, the end user, the model group and the token counts (Spend Tracking). For the cluster’s self-hosted models, the cost per token is configured with the €/token we manufactured above, and that is the hook between the cost model and the real accounting.

Tags for cost centres. The tags let you track spend and set budgets per label, categorising costs by project, department or cost centre (Setting Tag Budgets). A tag is attached when the key is created and every request made with that key inherits the tag automatically; the proxy applies the label’s budget. That way chargeback maps cleanly onto the organisation’s accounting hierarchy without every team having to remember to tag by hand.

A declarative catalogue fragment, which fits into the litellm-config of the router post, materialising the manufactured cost and two tenants:

litellm_settings:
 # €/token manufactured in this post for the self-hosted model.
 # input/output in €/token (not per 1M); 0.186 €/1M = 1.86e-7 €/token at u=80%.
 model_cost_map:
 llama-70b-onprem:
 input_cost_per_token: 0.000000186
 output_cost_per_token: 0.000000186

# Virtual keys per team (via the /key/generate API or config):
# equipo-datos: max_budget=500 €/month, tpm_limit=200000, tags=["cc-datos"]
# equipo-soporte: max_budget=150 €/month, tpm_limit=60000, tags=["cc-soporte"]
# The proxy counts tokens, multiplies by model_cost_map, deducts from the budget,
# and attributes the spend to the tag → showback per cost centre, without touching the engine.

The complete attribution flow, from the request to the per-team report:

Showback vs chargeback: the distinction the FinOps Foundation insists on

The two words are constantly confused and the difference is one of accounting formality, not technology:

Showback — giving each team visibility of what it consumed and its cost, without really billing it to their budget. The report arrives, the team sees it, but the money does not move between cost centres.
Chargeback — really transferring the cost to the team’s or product’s budget, putting it in their P&L. The spend stops being a central IT cost and becomes an allocated cost.

The FinOps Foundation is explicit on two points worth carving in stone (Invoicing & Chargeback, Chargeback vs Showback). First: showback is a requirement of any FinOps practice; chargeback depends on the organisation’s accounting policy. Not all organisations can or want to move money between departments for GPU consumption; showback you can always do. Second, and counterintuitive: neither of the two is “more mature” than the other. The narrative that chargeback is the “grown-up” version of showback is false according to the framework itself. The practical recommendation: start with showback, giving visibility, then build cost allocation aligned to the organisational hierarchy, and only then, if the accounting policy supports it, switch chargeback on.

For the on-premise GPU cluster this means: the LiteLLM machinery (keys, tags, spend tracking) always produces the showback. Turning it into chargeback is a decision for finance, not for the platform team. The platform team guarantees that the numbers are correct and reproducible; who really pays is a governance decision.

Isolation: making the cost genuinely attributable

Chargeback is only honest if consumption is isolable. If two tenants share a GPU without partitioning and one saturates the HBM, the other suffers degradation it did not cause but which contaminates its cost attribution. Isolation has two planes.

Hard GPU isolation: MIG. Multi-Instance GPU physically partitions an H100 into up to 7 instances with dedicated memory and SMs; in a 7-way partition each instance has ~10 GB of HBM3 and its own SMs. MIG gives the strongest isolation: tenant A in its MIG instance cannot touch tenant B’s performance, and attribution is trivial because each instance is accountable separately (see Sharing a GPU for the detail of MIG vs time-slicing vs MPS). The cost: MIG instances are fixed, they are not resized hot, and if they sit empty they are fragmented and wasted capex, the coworking desk subdivided into booths nobody rents.

Logical Kubernetes isolation. On top of the cluster:

Namespaces per tenant — the boundary for RBAC, NetworkPolicy and quotas.
ResourceQuota — limits how many nvidia.com/gpu (or nvidia.com/mig-1g.10gb) a namespace can request. It is the GPU capacity quota at the scheduler level: the tenant cannot claim more GPUs than its quota grants, which bounds its maximum cost by construction.
PriorityClass — defines which workloads can evict others under pressure. The overnight re-embeddings batch runs with low priority and gives way to interactive inference; that way it fills the utilisation valleys without risking the paying tenant’s SLO. It is the piece that stops “raising utilisation with batch” from cannibalising quality of service.

The combination that works in the generic cluster: MIG to partition the GPUs between tenants that need hard isolation and clean attribution, ResourceQuota per namespace to bound each tenant’s cost, and PriorityClass so that filler work raises utilisation without touching the priority tenants.

Applied to the generic 4×H100 cluster

Let us bring it all down to the series’ cluster: 4 nodes × 4×H100 SXM 80 GB, 16 GPUs, ≈ 17.1 €/hour of fixed cost (≈ 150,000 €/year). Three teams share it: Data (production RAG, office-hours traffic), Support (ticket assistant, daytime peaks) and Platform (evals and re-embeddings batch, no urgency).

Splitting the cost. The cluster’s fixed cost (150,000 €/year) is split through LiteLLM’s spend tracking in proportion to the tokens consumed by each team, valued at the manufactured €/token. If in one month Data consumed 1,700 M tokens, Support 480 M and Platform 320 M, at 0.186 €/1M the showback is ≈ 316 € / 89 € / 60 €. The residue, the fixed cost of the GPUs nobody used because average utilisation was, say, 45 %, is the cost of idleness, and the governance decision is whether it is split between the tenants (penalises everybody equally) or charged to Platform as “cost of unsold capacity” (gives Platform an incentive to raise utilisation). The FinOps Foundation would say: make it visible first (showback of the cost of idleness), decide the split afterwards.

Quotas per tenant. Each team has its namespace with ResourceQuota: Data can claim up to 8 GPUs (2 replicas at TP=4), Support up to 4, Platform up to 4 but with a low PriorityClass, giving up its GPUs when Data or Support need them at peak. In LiteLLM, each team has its virtual key with a monthly max_budget and a tpm_limit that reflects its capacity quota.

MIG to isolate and attribute. For the small models (embeddings, reranker, a utility Llama 8B), an H100 split 7-way with MIG gives seven separately attributable instances: three for Data’s embeddings, two for Support’s reranker, two free for filler. Each instance is an independent accounting “desk”; the per-tenant showback comes straight from the scheduler.

The governance metric is utilisation, not the €/token. Here we close the circle. The KPI the cluster owner should look at daily is not “what the token costs”, that number only falls as a consequence, but DCGM_FI_DEV_GPU_UTIL, the DCGM exporter metric that indicates what fraction of the time the GPU is not idle (DCGM exporter). And with a critical nuance that already appeared in capacity planning: DCGM_FI_DEV_GPU_UTIL measures “not idle”, it does not measure useful work. A GPU can show 100 % GPU-util with the HBM saturated and low SM occupancy, producing few tokens. That is why serious FinOps crosses three signals: DCGM_FI_DEV_GPU_UTIL (is the GPU busy?), DCGM_FI_PROF_SM_OCCUPANCY and DCGM_FI_PROF_GR_ENGINE_ACTIVE (is it doing real work?), and vllm:gpu_cache_usage_perc (is the HBM well used?). The economic governance metric is useful tokens per GPU-hour, and it is watched from the GPU observability with DCGM dashboard.

An end-to-end assembly detail, LibreChat on top of LiteLLM with RAG, which closes the flow from tenant to cost, is covered in the (forthcoming) post on the end-to-end sovereign assistant; here it is enough to know that the metering point is always the proxy, never the engine.

Four traps of on-premise GPU FinOps

Trap 1 — comparing the on-premise €/token with the commercial API’s list price. The commercial API has margins, SLA and scale the on-premise cluster does not, but on-premise has the cost of idleness the API hides from you (they fill their GPUs with thousands of customers). The honest comparison is on-premise at its real utilisation vs the API at its real negotiated price, not the fantasy of on-premise at 100 %.

Trap 2 — forgetting the cost of idleness in the showback. If you only charge teams for the tokens they consumed, the cost of the idle GPUs disappears from the report and nobody sees it. That cost exists and somebody is paying it. Making it visible is the first step to reducing it.

Trap 3 — confusing high GPU-util with efficiency. A GPU at 100 % DCGM_FI_DEV_GPU_UTIL with the HBM saturated and low SM occupancy spends capex without producing proportional tokens. The goal is not “GPU at 100 %”, it is “useful tokens per euro”. Always cross util with SM occupancy and real throughput.

Trap 4 — chargeback before showback. Switching chargeback on, moving money, before the teams trust that the numbers are correct generates disputes that burn the whole FinOps programme. First visibility, then trust in the data, then, if the accounting policy allows it, the money.

References

NVIDIA — H100 Tensor Core GPU Datasheet (TDP 700 W, HBM3 3,35 TB/s): resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306.
LiteLLM — Virtual Keys (docs.litellm.ai/docs/proxy/virtual_keys), Budgets & Rate Limits (/docs/proxy/users), Spend Tracking (/docs/proxy/cost_tracking), Setting Tag Budgets (/docs/proxy/tag_budgets), Team Budgets (/docs/proxy/team_budgets).
FinOps Foundation — Invoicing & Chargeback Capability (finops.org/framework/capabilities/invoicing-chargeback/) y Data Analysis and Showback (finops.org/framework/previous-capabilities/analysis-showback/).
CloudZero — Chargeback vs. Showback: Cloud Cost Allocation Models Explained (cloudzero.com/blog/chargeback-vs-showback/).
NVIDIA — DCGM Exporter (docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_GR_ENGINE_ACTIVE.

Parameter	Symbol	Example
Requests per day (average)	\(D\)	5,000,000
Input tokens per request (average)	\(L_{\text{in}}\)	512
Output tokens per request (average)	\(L_{\text{out}}\)	256
Peak factor (peak hour vs average ratio)	\(k_{\text{peak}}\)	1.8
Hours per day with significant traffic	\(H\)	16

Finops on lo0 — Tech Blog

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

TL;DR

You are here: the seam between DEPLOY and OBSERVE

The analogy: the toll booth that issues the ticket

Three routes, and one with an expiry date

The configuration that works today

What is lost when moving to OTel v2

Cost: why the trace says zero

Registering the price

How that number reaches Langfuse, and who wins

The tokens of a streaming response

Correlation: from the client to the engine

What does work

What does not work without effort

The vLLM side, and why the Collector stops being optional

The four serial queues

What cannot be promised to the auditor

PII: switching off the content before it leaves

The price in latency

Start-up checklist

Traps and things that are not what they look like

Closing

See also

Sources

From public cloud to private: what you really lose and which projects fill the gap

TL;DR

The analogy: leaving the hotel to set up your own house

Why this conversation belongs to 2026, not to always

What you lose when you leave the cloud, specifically

The map of projects that fill each gap

The gap you will not close

When NOT to repatriate

The numbers, to ground it

For your sovereign inference factory

See also

Sources

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

TL;DR

1. The sizing chain: from demand to the number of GPUs

1.1 Step 1 — Characterise the demand

1.2 Step 2 — Set the SLO and derive the maximum utilisation

1.3 Step 3 — Required throughput and throughput per GPU

1.4 Step 4 — Number of GPUs and number of nodes

1.5 Summary of the chain

2. From sizing to cost: the TCO model

2.1 Annual cost per node (base scenario)

2.2 From €/node-year to €/GPU-hour

2.3 From €/GPU-hour to €/1M tokens

2.4 Comparison with cloud (equivalent €/1M tokens)

3. Break-even and payback of your own investment

3.1 The utilisation break-even formula

3.2 Capex payback

3.3 Simple payback (capex only vs equivalent cloud)

4. Decision table: when to buy, rent or go hybrid

5. Sensitivity analysis

5.1 Sizing vs hourly profile and peak factor

5.2 Break-even vs sustained utilisation

5.3 Break-even vs energy price

5.4 Break-even vs demand growth

5.5 Headroom: the cost of the safety margin

6. Sensitivity map: €/GPU-hour and break-even on two axes

7. Integration with the rest of the FinOps chain

Sources

Full TCO of an on-premise GPU cluster: from capex to the all-in €/GPU-hour and the break-even against cloud

TL;DR

The model: declared assumptions

Capex breakdown per 4×H100 SXM5 node

GPUs

Server / HGX baseboard

InfiniBand NDR network

NVMe storage

Rack, PDU and datacenter connectivity

Capex summary per 4×H100 SXM5 node

Opex breakdown per 4×H100 SXM5 node (annual)

Energy

Staff / operations

Maintenance and support

Depreciation (for accounting purposes)

Annual opex summary per 4×H100 SXM5 node (base scenario, 8-node cluster)

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`