Cost-per-Token on lo0 — Tech Blog

The second cost vector of AI agents: durable execution with Temporal

Sun, 14 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

The thesis: agents have two cost vectors, not one

Almost all the cost discourse around agentic AI focuses on a single vector: the cost of inference (the tokens). And that one is solvable: FastMCP + vLLM on local hardware eliminates the cost of inference — you serve the models on your own iron, the cost per token drops to your amortised cost, with no API bill. But there is a second vector that almost nobody measures and that kills more projects: the cost of faulty execution. An agent is not a call, it is a sequence of steps (reason, call a tool, reason, call another one, and so on), and when that sequence fails halfway through with no persisted state, the cost is not linear: it doubles, it spirals with retries and it inflates on every recovery. This article is about that second vector, how durable execution attacks it (with Temporal as the reference), what the state of the art and the papers say, and why it is the real cause of so many cancellations.

The framing that backs the urgency comes from Gartner: it predicts that more than 40 % of agentic AI projects will be cancelled before the end of 2027, because of rising costs, unclear business value or inadequate risk controls (Gartner). “Rising costs” is exactly this second vector: not the cost of serving the model, but the cost of running it badly.

The three expenses of a faulty agent

The cost of faulty execution breaks down into three concrete sources, and it is worth seeing them with numbers:

Vector 1: failures charged twice

With no persisted state, a failure at step 15 of 20 costs you the tokens of the 14 previous steps (which you already paid for) plus the full retry from scratch. The wasted cost of an agent with no checkpoint, per failure, is:

$$\text{wasted cost} = \text{tokens of the completed steps} \times \text{cost per token} \times (\text{retries})$$

The longer the sequence and the later it fails, the more expensive: a failure at 75 % of the work throws away 75 % of the cost. Durable execution checkpoints every step: if the worker dies halfway through, another one picks the workflow up exactly where it stopped, not from scratch (Temporal). Temporal uses its Event History as the record of past decisions; on a failure it replays the progress to date and resumes at the exact point (CallSphere). Step 15 fails → step 15 is retried, not the previous 14. The wasted cost goes from “the 14 steps” to “the step that failed”.

Vector 2: uncontrolled retries

Agents that retry with no policy blow up the API or compute bill: a retry loop against a rate limit can generate dozens of billable calls for a single task. Durable execution defines retry policies per activity: hard limits on retries, exponential backoff and timeouts that bound the maximum possible cost of each call. Instead of “retry until it works” (unbounded cost), you define “at most N retries with backoff and a timeout of T” (bounded cost).

The state of the art backs this with data: the ReliabilityBench benchmark evaluates agent reliability under controlled failures and finds that rate limiting is the most damaging failure of all (arXiv 2601.06112). That is, exactly the failure that an unpolicied retry turns into a cost spiral. Bounding retries is not hygiene, it is the defence against the failure mode the literature singles out as the worst.

Vector 3: inflated context on recovery

With no persisted state, the agent rebuilds the full context on every recovery: it reloads the whole conversation/task history to “remember” where it was, paying the tokens of that enormous context on every recovery. With durable execution the state lives in the workflow history, so the next step receives only the context needed for that step, not the whole conversation. The saving is direct: fewer input tokens per step, and no penalty for rebuilding the context after each failure.

A worked example: the cost of one failure

Let us put numbers on the three vectors. A 20-step agent consuming ~3,000 tokens per step (reasoning plus tool call), on in-house inference at ~1.09 €/1M tokens, with a failure at step 15 and a failure rate of 10 % of executions:

Scenario	Tokens recomputed per failure	Cost per failure	Over 10,000 executions (10 % fail)
No state (full retry)	14 steps × 3,000 = 42,000	~0.046 €	~46 € wasted
With checkpoint (resumes at 15)	0 (only step 15 is retried)	~0.003 €	~3 € wasted

The absolute figure looks small because inference is in-house and cheap; but the ratio is ~15× less waste per failure, and it scales with the length of the sequence and with the cost per token (were it an external API at 10-30 €/1M, the waste without a checkpoint would be multiplied by 10-30). And this is only vector 1: add uncontrolled retries (vector 2), which can turn one failure into dozens of calls, and the context rebuilt on every recovery (vector 3). The cost of faulty execution is not the inference cost × something small; it is a multiplier that grows with the agent’s complexity, and it is exactly what does not show up in the demo and does show up in the production bill.

How Temporal works: workflows, activities and determinism

To understand why Temporal achieves this, its two primitives:

Workflow: the agent’s orchestration logic (the sequence of steps, the decisions). It has to be deterministic: given the same Event History, it produces the same decisions. That is what allows the execution to be replayed after a failure and to arrive at the same point.
Activity: every step with side effects (a call to the LLM, to a tool, to an API). Activities are not deterministic (the LLM is probabilistic), so their result is persisted in the Event History: on replay they are not executed again, they are read from the history.

The trick is that separation: the deterministic logic (workflow) is replayed cheaply from the history; the expensive, non-deterministic steps (activities, the model calls) are persisted and not repeated. That is why “completed work is never repeated”: completed activities are read, not recomputed. And retry policies are defined per activity, so each call to the model or to a tool has its own retry limit, backoff and timeout, which is the vector 2 cost control at the right level.

Durable execution: what it is and the landscape

Durable execution is a programming model that guarantees the code completes despite failures: every step is checkpointed automatically, and if a worker dies, another one picks the workflow up where it stopped (Temporal). It crossed into the “early majority” in 2025, with offerings from AWS, Cloudflare and Vercel, driven precisely by AI agent infrastructure (Inngest).

The tooling landscape (the 2026 reference architecture for durable LLM agents): Temporal, AWS Step Functions Express, Restate, DBOS and Inngest as durable execution primitives; and the checkpointer model of LangGraph (PostgresSaver, RedisSaver, DynamoDBSaver) on the agent framework side (render.com). Temporal stands out for its workflow-as-code model and its Event History as the source of truth.

Option	Model	Self-hosted	Note for sovereignty
Temporal	workflow-as-code, Event History	yes (open source)	the most mature; full on-prem
Restate	durable RPC/handlers	yes	lightweight, modern
DBOS	durable on top of Postgres	yes	state in your own Postgres
Inngest	event-driven, steps	SaaS / self-host	focus on DX
AWS Step Functions Express	state machine	no (AWS SaaS)	tied to AWS, not sovereign
LangGraph checkpointer	framework checkpointer	yes (your backend)	at framework level, not platform level

For a sovereign platform, the ones that are self-hosted (Temporal, Restate, DBOS) keep state in your cluster; the SaaS ones (Step Functions) move execution into someone else’s jurisdiction. Temporal is the choice for maturity and for its complete on-prem deployment.

The cost of durable execution itself

Honesty about the data: durable execution is not free. It has two costs to watch:

Execution overhead: persisting state and replaying history adds latency and compute. The reference architectures put it at around 5-20 % overhead on a load of 100,000 executions/day (render.com).
Price per step: in the offerings with per-step pricing, orchestrating many model calls, rate limit retries and complex interactions generates a large number of billable steps, and the cost can run away (render.com).

The conclusion for a sovereign platform: self-host the durable execution engine (Temporal is open source and deploys on-prem) and the SaaS per-step price disappears, leaving only the execution overhead, a 5-20 % that is paid back many times over by what it saves on the three vectors above. The arithmetic: if a failure halfway through a sequence used to cost you 50-75 % of the recomputed cost, a 5-20 % overhead is a bargain.

Observability: from manual post-mortem to real-time traceability

The fourth element of the equation: the observability layer. The integration with Braintrust and the complete history of every execution (Temporal’s Event History) turn cost analysis from a manual post-mortem, reconstructing by hand what happened and what it cost after a failure, into real-time traceability: every execution is recorded step by step, with its tokens, its retries and its cost, ready to analyse. This closes the FinOps loop of the cost-per-token article, but at the agent level: not just how much a token costs, but how much an execution costs, and where it is wasted.

The complete equation

Putting the pieces together, the cost architecture of a sovereign agent:

FastMCP + vLLM running locally eliminates the cost of inference. Temporal eliminates the cost of faulty execution. They are complementary, attacking different vectors of the same problem. Whoever solves only the first (serves cheaply but executes badly) keeps bleeding through the second; whoever solves only the second (executes robustly but pays for the API) keeps bleeding through the first. A sustainable platform needs both.

State of the art and papers

The reader asked for papers, and there are some: agent reliability and cost is an active research area in 2026.

Paper	Contribution
ReliabilityBench (2601.06112)	an agent reliability benchmark: consistency, robustness and tolerance to tool/API failures; rate limiting is the most damaging failure; ReAct more robust than Reflexion under stress
Cost-Efficient LLM Agents (arXiv)	the reliability-cost trade-off: routing every decision through the LLM improves accuracy but blows up cost; pre-coded graphs are cheaper but brittle against compound failures
Multi-agent Byzantine Fault Tolerance (2511.10400)	BFT consensus (CP-WBFT) to stabilise multi-agent systems under extreme failure rates (85.7 %)
ACRFence (2603.20625)	security of agent checkpoint-restore: preventing semantic rollback attacks
The Six Sigma Agent (2601.22290)	enterprise reliability via consensus-driven decomposed execution

Two readings for the proposal. First, the reliability-cost trade-off is real and is characterised: you cannot maximise accuracy ignoring cost, or the other way round; durable execution shifts the frontier by making reliability cheaper (checkpoint plus bounded retry instead of recomputing). Second, agent checkpoint-restore has its own security surface (ACRFence): persisting state introduces rollback risks that have to be controlled — durability is not only a cost decision, it is a security one too.

The sovereign angle

For a European sovereign platform the combination fits perfectly and is entirely on-prem: vLLM serves the models (no API cost and no sovereignty given away), FastMCP exposes the tools, and self-hosted Temporal orchestrates with durability (no SaaS per-step price and no execution data leaving for a foreign jurisdiction). The execution history, which describes in detail what your agent does and with which data, stays in your cluster, just like the inferences. Against a stack of agents on external APIs plus a US SaaS orchestrator, the sovereign stack removes both cost vectors and keeps the data under EU jurisdiction. It is the same argument as the rest of the series, applied to the agent layer: bounded, measured and sovereign cost.

FastMCP and the tool layer

The other component of the equation is FastMCP, the layer that exposes tools to the agent via MCP (Model Context Protocol). The agent does not call the APIs directly: it calls MCP servers that encapsulate each tool (a database, a search engine, an internal system) with a uniform contract. The connection with cost and durability: every call to an MCP tool is a Temporal activity, so it inherits its retry policy, its timeout and its persistence. That is, FastMCP defines which tools exist and Temporal governs how they are executed with durability and bounded cost. And because the MCP servers are your own, the tools and the data they touch stay on-prem, never exposed to an external provider. The triad vLLM (model) + FastMCP (tools) + Temporal (orchestration) is a fully sovereign agent stack: the model, the tools and the execution, all under your control and with the cost of both vectors bounded.

Patterns: human-in-the-loop and long sequences

Where durable execution shines most, by agent pattern:

Human-in-the-loop (HITL): human approval patterns, essential for an agent’s safety, map directly to the suspend/resume primitives of durable execution, allowing workflows that pause for hours or days waiting for an approval without losing state (Inngest). Without durability, keeping an agent “waiting” consumes resources or loses the context; with it, the workflow sleeps at no cost and wakes up with all its state.
Long sequences: the longer the chain of steps, the greater the checkpoint’s saving (a late failure throws away more work). Multi-step agents with tools are the canonical use case.
Multi-agent: when several agents collaborate, reliability gets complicated (the BFT papers study this); durable execution provides the consistent state substrate on which to build that coordination.

The pattern that does not need it: a single-step, stateless call (a classifier, a simple extraction). There the durability overhead buys nothing. The rule: durable execution for stateful sequences that fail halfway through; for everything else, a direct call.

How to measure the second vector

To manage the cost of faulty execution you have to measure it, and with the Event History plus observability (Braintrust) the KPIs are within reach:

KPI	What it indicates	Where it comes from
Tokens wasted per failure	the cost of vector 1	tokens recomputed after a failure
Retries per activity	the cost of vector 2	retry count from the Event History
Context size on recovery	the cost of vector 3	input tokens after a recovery
Failure rate per workflow	how often the problem occurs	failed executions / total
Cost per complete execution	the business figure	sum of tokens × cost/token per run

The last one closes the circle with cost per token: not just “how much a token costs”, but “how much an agent execution costs, and how much of that was useful work versus waste”. Without these KPIs the second vector is invisible, and what is invisible does not get optimised, it gets cancelled when the bill surprises you. With them, the cost of every execution is a real-time figure, and the decision to make each workflow durable (or not) is taken with numbers.

Limits and traps (data-driven)

Durability is not free. 5-20 % overhead; on SaaS with per-step pricing it can run away. Self-host Temporal to remove that price.
Not every agent needs it. A single-step, stateless task gains nothing from durable execution; the value is in the long sequences that fail halfway through.
Bounded retries can hide real failures. A retry limit avoids the cost spiral, but you have to alert when it is exhausted, not swallow the failure in silence.
Checkpoint = security surface. Persisting state opens the door to rollback attacks (ACRFence); protect it.
The first vector still exists. Temporal does not lower the inference cost; you need local vLLM for that. They are complementary, not substitutes.

The conclusion the data and the papers support: the cost of agentic AI has two vectors, and the industry has obsessed over the first (inference) while ignoring the second (faulty execution), which is precisely where Gartner locates the cause of 40 % of cancellations. FastMCP + vLLM solves the first; Temporal the second. A sovereign and sustainable agent platform needs to attack both, with observability that turns the cost of each execution into a real-time figure rather than an autopsy.

Closing

The dominant narrative about agent cost is half-lame: it treats the inference cost as if it were the only one, when it is only the one you see in the demo. The one that kills projects, the one Gartner files under “rising costs” inside its 40 % of cancellations, is the cost of faulty execution: failures charged twice, retries with no ceiling and the context rebuilt on every recovery. The engineering that attacks it is not a prompt trick or a better model, it is an execution model: durable execution, which checkpoints every step, bounds every retry and passes only the necessary context. Temporal, open source and self-hosted, does this while keeping state in your cluster, and combined with vLLM (model) and FastMCP (tools) it composes a fully sovereign agent stack where both cost vectors are bounded and the data never leaves the EU. The state of the art and the papers confirm the problem is real and characterisable: the reliability-cost trade-off, rate limiting as the worst failure, checkpoint-restore as a new security surface. For anyone who wants to be in the 60 % of agentic projects that are not cancelled, the recipe is the one in this article: solve both vectors, not one, and measure the second as seriously as the first.

Sources

Gartner · más del 40 % de proyectos de IA agéntica cancelados antes de 2027 — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Temporal · durable execution — https://temporal.io/
CallSphere · Temporal para workflows de agentes (Event History, replay) — https://callsphere.ai/blog/temporal-ai-agent-workflows-durable-execution-workflow-as-code
Inngest · durable execution, clave para agentes en producción — https://www.inngest.com/blog/durable-execution-key-to-harnessing-ai-agents
render.com · plataformas de workflow durables para agentes/LLM (overhead, precio por paso) — https://render.com/articles/durable-workflow-platforms-ai-agents-llm-workloads
ReliabilityBench (arXiv 2601.06112) — https://arxiv.org/abs/2601.06112
Cost-Efficient LLM Agents (arXiv 2603.01548) — https://arxiv.org/pdf/2603.01548
Byzantine Fault Tolerance multi-agente (arXiv 2511.10400) — https://arxiv.org/abs/2511.10400
ACRFence · checkpoint-restore de agentes (arXiv 2603.20625) — https://arxiv.org/pdf/2603.20625
The Six Sigma Agent (arXiv 2601.22290) — https://arxiv.org/html/2601.22290

From GPU-hour to cost per token: the metric that compares on-prem and cloud

Sun, 14 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Fourth article in the FinOps track (A4), and the one that closes out the metric that gives the whole pillar its meaning: cost per token. In A2 we saw how OpenCost assigns the cost of the iron (the pod, in €/hour); in A3, which tool to use. But the cost of the pod is not the metric that compares on-prem with cloud, nor the one the business understands: that one is cost per token (or per request). This article explains how it is really calculated (spoiler: for your own inference it is not a list price, it is €/GPU-hour divided by throughput), how to instrument it with a gateway (LiteLLM), how to attribute it by team and product, and how to compare it against European cloud. No recommendations; only the mechanics and the numbers.

The identity: from GPU-hour to token

Cost per million tokens (CPM) is the cost of the iron divided by what it produces:

$$\text{cost per token} = \frac{\text{cost of the iron (€/h)}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

This is the central formula of the FinOps track, and it has a consequence many people never quite internalise: for a self-hosted model, the cost of a token does not exist as a fixed price. It depends on two things that change: the cost of the iron (from OpenCost, in €/hour) and the throughput (from your benchmark, in tok/s). The same model on the same node costs half as much per token if you double the throughput with an optimisation, because the denominator changes, not the numerator. The cost per token of your own inference is a function of your efficiency, not a tariff.

That sets it radically apart from an external API, where the price per token really is a fixed tariff the provider charges you. Comparing the two requires turning your variable cost into a number, and that is where the iron (A2) meets the tokens (this article).

The two halves of cost per token

Cost per token comes from joining two measurements that live in different places:

The iron half comes from OpenCost (article A2): the by-usage cost of the vLLM pod, in €/hour. The token half comes from the gateway, which intercepts every request and counts tokens per model and per team. Divide, and out comes the cost per token and, crucially, per team, which is what enables chargeback.

The gateway: LiteLLM in depth

The gateway sits between the application and the engine (vLLM) or the provider, and intercepts every request. LiteLLM is the OSS reference: it proxies over 100 models and tracks spend by keys, users and teams, logging spend for all known models automatically (LiteLLM · Spend Tracking).

Its key mechanisms:

Mechanism	What it does
`response_cost`	the cost of each response, available in logging (`kwargs["response_cost"]`)
Spend logs	one row per request with tokens, model, user, key and cost (SQLite/PostgreSQL)
Tags	tag each request by team, project, environment and model tier
Budgets	budgets and rate limits per team or user
Custom callbacks	bespoke loggers for key/user/model/tokens/cost

With the spend logs table (via the database adapter) you have the detail of every request to build cost dashboards (LiteLLM · Logging), and you can tag each request with team/project/environment for per-team analysis, with notifications (Slack) and hard caps for keys that must not exceed a limit. In practice it is the platform’s token meter and cost allocator.

The self-hosted trap: LiteLLM does not know what your own token costs you

Here is the point almost nobody wires up correctly. LiteLLM tracks cost using list prices for known models (the API ones: OpenAI, Anthropic and so on). But for a model self-hosted on your vLLM there is no list price, because you pay nobody per token. If you tell it nothing, LiteLLM does not know the real cost of your own token. The solution is custom pricing: declaring to LiteLLM the cost per token of your self-hosted model (LiteLLM · Custom Pricing).

And where does that number come from? From the identity above: the cost of the iron (OpenCost) divided by the throughput (your benchmark). That is, the correct flow is:

OpenCost gives you the cost of the vLLM pod: ~11 €/hour (A2).
Your benchmark gives you the throughput: ~2,800 tok/s (track B).
The identity gives the cost per token: 11 ÷ (2,800 × 3,600 / 10⁶) ≈ 1.09 €/1M tok.
That 1.09 €/1M goes into LiteLLM as custom pricing.

Only then does the response_cost of each request reflect the real cost of your inference, rather than a zero or somebody else’s list price. Without this step, your LiteLLM cost dashboards lie about your own models. It is the explicit junction of the three tracks: iron (FinOps), throughput (benchmarking) and measurement (gateway).

Configuring custom pricing: an example

In LiteLLM, the real cost of a self-hosted model is declared in its configuration. An example with the cost per token derived from the identity (1.09 €/1M ÷ 10⁶ = 0.00000109 €/token):

# LiteLLM config.yaml — real cost of a self-hosted model
model_list:
 - model_name: llama-3.1-70b-onprem
 litellm_params:
 model: openai/llama-3.1-70b  # your vLLM endpoint
 api_base: http://vllm:8000/v1
 input_cost_per_token: 0.00000109 # €/token (from OpenCost ÷ throughput)
 output_cost_per_token: 0.00000109 # refine if you separate prefill/decode

With that, every response_cost LiteLLM records reflects the real cost of your iron. The operational note: this number must be updated when the cost of the node changes (new depreciation, a different energy price) or the throughput does (an engine optimisation). It is not a constant; it is the output of OpenCost and of the benchmark, and it is recalculated whenever either of the two changes. Automating that recalculation, from node cost and measured throughput to custom pricing, is what keeps cost dashboards honest.

Cost per token changes with throughput: read it off the sweep

Because throughput is the denominator, cost per token is not a point, it is a curve over the concurrency sweep. On the ~11 €/hour node:

Concurrency	Throughput (tok/s)	Cost per 1M tokens
1	350	~8.73 €
8	2,100	~1.46 €
16	3,330 (goodput)	~0.92 €
24	3,900 raw / 2,420 goodput	~0.78 € raw / ~1.26 € on goodput

Important readings: at low concurrency the cost per token is sky-high (the fixed cost of the node is shared among few tokens); cost falls as occupancy rises. But watch the last row: if you compute cost on raw throughput (3,900) you get 0.78 €, whereas if you compute it on goodput (2,420, the part that meets the SLO) you get 1.26 €, and the latter is the real cost, because requests that violate the SLO do not count as useful production. An honest cost per token is computed on goodput, not on catalogue throughput. This is where the cost track meets the benchmarking one: the correct denominator is goodput.

Cost per request: not only per token

Cost per token is the unit; cost per request is what a complete request bills. A request consumes input tokens (prompt) plus output tokens (generation), and both count:

$$\text{cost per request} = (\text{input tokens} + \text{output tokens}) \times \text{cost per token}$$

With nuances: in many engines the prefill (input) and the decode (output) have different costs per token (decode is memory-bound and more expensive per token), so a fine-grained cost model distinguishes the two. For most cases a weighted average cost per token will do; to optimise, it pays to separate prefill and decode. LiteLLM records input and output tokens separately in its spend logs, so the data is there to refine with.

Cost per request is what connects to the product: if a feature consumes 1,500 tokens on average (1,200 of prompt plus 300 of output) and the cost is 1.09 €/1M, each use costs ~0.0016 €, the basis for setting margin, as seen in the FinOps introduction.

Cost per model and per tier

A platform does not serve a single model. Cost per token varies by model and by configuration, and the gateway tells them apart. An 8B and a 70B on the same node have very different costs per token: the 8B yields far more throughput (more tokens/s) for the same GPU-hour, so its cost per token is a fraction of the 70B’s. The design consequence: routing each request to the cheapest model that meets the quality bar is a direct cost lever, since serving a 70B where an 8B suffices means overpaying per token.

Model (example, same node)	Relative throughput	Relative cost per token
8B	high	low
70B FP8	medium	medium
70B FP16	low	high

The gateway, with its per-model custom pricing, makes this difference visible and enables cost-based routing: classify the request and send it to the appropriate tier. It connects with the L7 router of the serving track: the gateway not only measures cost, it can decide on the basis of it. And quantisation (FP16→FP8) shows up again as a lever: the same model in FP8 raises throughput and lowers cost per token, moving the figure without changing model.

Attribution: from cost to team and product

The reason for putting a gateway in place is not only to measure total cost, it is to share it out. By tagging each request with team, product and environment, LiteLLM builds the split by business dimension:

With per-team budgets and hard caps per key, the gateway does not only report: it governs. A team that goes over budget gets an alert; a key with a hard cap cannot spend any more. It is the FinOps Operate phase applied to the token.

A combined report example

Joining the iron (OpenCost) and the tokens (gateway), the monthly cost-per-token-per-team report on the example node:

Team	Tokens/month	Applied cost per token	Charged cost
A · chat-prod	240M	1.09 €/1M (70B FP8)	~262 €
B · batch	90M	0.40 €/1M (8B)	~36 €
C · experimentation	15M	1.09 €/1M (70B)	~16 € + idle

What it reveals: A consumes a lot but at an efficient cost (FP8, high utilisation); B uses a cheap model (8B) for its workload; C consumes little but, as we saw in A2, drags along the idle cost of its underused GPU. The token report plus the OpenCost iron report give the full picture: who spends, on which model, with what efficiency. It is the chargeback that neither half would produce alone. And the per-team figure is what gets taken into the budget conversation: not “the cluster costs X”, but “your team consumed Y million tokens at Z €/million”.

Governing spend: budgets, caps and alerts

Measuring without governing is a dashboard; the value lies in closing the loop. LiteLLM allows three levels of control over spend:

Control	What it does	When
Soft budget	alert (Slack) when a threshold is crossed	tracking teams
Rate limit	limits requests/tokens per minute	avoiding cost spikes
Hard cap	the key cannot spend any more	keys that must never overrun

The healthy pattern: soft budgets with alerts for the teams (so they adjust their consumption), rate limits to smooth spikes, and hard caps only on critical keys (a service that must not run away). With this, cost per token stops being an after-the-fact figure and becomes a real-time control: the FinOps Operate phase, applied to inference.

On-prem vs European cloud comparison (the reason for all of it)

Cost per token exists to answer one question: is it cheaper to serve on my own iron or to rent? With the identity and European figures:

Option	Cost per 1M tokens	Notes
Depreciated on-prem (8×H100, 2,800 tok/s)	~1.09 €	your iron, your sovereignty
European cloud Scaleway (8×H100, on-demand)	~2.17 €	EU, no iron to operate
External API (proprietary model)	provider tariff	no data sovereignty

Cost per token is what makes the incomparable comparable: your variable cost (iron ÷ throughput) against a provider’s fixed tariff. And it exposes the lever: if an optimisation raises your throughput from 2,800 to 4,200 tok/s, your on-prem cost drops to ~0.73 €/1M, widening the advantage over renting. That is why benchmarking (track B) is, indirectly, the tool that moves cost per token the most: every goodput improvement translates into this number.

From cost per token to product price

Cost per token closes the loop with the business when it becomes cost per unit of product. The chain: cost per token (from the identity) × tokens per use of a feature = marginal cost of that feature. With that, three business decisions become data-backed:

Setting a price. If serving a feature costs 0.0016 € per use, the price to the customer has to cover it with margin; cost per token is the floor of pricing.
Spotting loss-making features. A feature that consumes many tokens (long prompts, lots of output) may cost more than it contributes; cost per request reveals it before you scale.
Comparing build vs buy at product level. On-prem cost per token against an API tariff, for that specific workload, decides whether to serve in house or consume externally.

This is the ultimate reason for the whole FinOps track: it is not infrastructure accounting, it is the quantitative basis for product and architecture decisions. A measured and attributed cost per token turns “is this profitable?” from an intuition into a calculation.

Detailed comparison: on-prem, European cloud and API

Extending the comparison with scenarios, for the same job (1M tokens):

Option	Cost/1M tok	Sovereignty	Where it fits
Depreciated on-prem, high utilisation	~0.7–1.1 €	full (EU)	high, sustained volume
Depreciated on-prem, low utilisation	~2–4 €	full (EU)	bad: idle drives cost up
European cloud (Scaleway, on-demand)	~2.2 €	EU	variable volume, no iron to operate
European cloud (reserved/committed)	~1.5–1.8 €	EU	medium committed volume
External proprietary API	provider tariff	non-EU	prototype / no sovereignty requirement

The table shows two things. First, on-prem only wins if utilisation is high, since at low occupancy idle makes it dearer than cloud, which connects cost per token with scheduling and utilisation (the idle of A2). Second, the external API, however competitive its tariff, loses data sovereignty, an axis that for GDPR data is not negotiable. Cost per token is the metric that puts the first three on the same scale; sovereignty is the constraint that rules out the last one for certain kinds of data.

Prompt caching and effective cost

One nuance refines the real cost per token: prefix caching. When many requests share the same prefix (a long system prompt, a repeated context), the engine caches its prefill compute and does not recompute it on every request. That means cached input tokens cost almost zero the second time and thereafter, so the effective cost per token of a workload with a high cache hit rate is lower than the nominal cost. External APIs already reflect this with a reduced price for cached tokens; in self-hosted the effect is that throughput rises (less prefill to recompute), which by the identity lowers cost per token.

The implication for the cost model: if your workload has shared prefixes (RAG over the same corpus, agents with the same system prompt), the real cost per token is lower than the naive calculation gives, and it pays to measure the prefix caching hit rate so as not to overestimate cost. And the reverse: a workload of unique, never-repeated prompts gains nothing from the cache, and its cost per token is the full nominal figure. Cost per token, once again, is not a constant: it depends on the model, the precision, the utilisation, the goodput and the reuse pattern of the workload. Measuring it on your real traffic, not on a synthetic benchmark without a cache, is what gives the number you actually pay.

State of the art 2026

The gateway as standard: LiteLLM (and alternatives) are consolidating as the layer that measures tokens, attributes cost and governs budgets in the LLM stack.
FOCUS for tokens: layers on top of the gateway generate FOCUS-compatible logs, integrating AI cost with the rest of cloud spend (FinOps introduction).
Custom pricing as a bridge: the OpenCost→gateway connection via custom pricing is the standard way to give self-hosted models a real cost.
Cost per token as a business KPI: increasingly, cost per token (and per product) is the metric the business demands, not the cost of the pod.

Limits and traps (data-driven)

Forgotten custom pricing. Without declaring to LiteLLM the real cost of your self-hosted token, the response_cost of your own models is zero or someone else’s price. Close the OpenCost→gateway bridge.
Static throughput. Cost per token changes with throughput; if you optimise the engine but do not update the custom pricing, the reported cost drifts out of date.
Forgetting prefill/decode. An average cost per token is fine for reporting; to optimise, separate input and output.
Measuring without governing. Spend logs without budgets or caps make a pretty dashboard; the value lies in closing the loop with alerts and caps.
Comparing tariffs with costs without matching assumptions. Your 1.09 €/1M assumes a utilisation and a throughput; the provider’s tariff does not. Fix the assumptions before comparing.

With cost per token solved, the FinOps track has its central metric; the following articles (A5 multi-tenant chargeback, A8 TCO model) build on it. The full chain, iron (OpenCost) ÷ throughput (benchmark) = cost per token, measured and attributed by the gateway, is what turns the cost of AI into a defensible number, in euros and per team.

Closing

Cost per token is the metric that reconciles the three tracks of the series into a single number, and its underlying lesson is counter-intuitive: for your own inference, the cost of a token is not a price, it is the reflection of your efficiency. It comes from dividing the cost of the iron (OpenCost) by useful throughput (the goodput of your benchmark), and it becomes visible and governable with a gateway (LiteLLM) that counts the tokens, attributes them by team and applies budgets. The mistake that invalidates the whole exercise is leaving custom pricing at zero and believing your own inference is “free”; the right move is to close the OpenCost→gateway bridge and keep the number up to date when the node cost or the throughput changes. Done that way, cost per token answers with data the one question the business keeps repeating, is it worth serving this on my own iron, and it does so in euros, per team and per product, comparable against European cloud and against any API. For a sovereign architecture proposal, it is the column of the dashboard that translates all the cost engineering into the sentence that closes a budget meeting: this is the cost per token, this is how it is measured, and here is the bench to reproduce it.

Sources

LiteLLM · Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM · Custom Pricing (modelos self-hosted) — https://docs.litellm.ai/docs/proxy/custom_pricing
LiteLLM · Logging — https://docs.litellm.ai/docs/proxy/logging
LiteLLM · documentación general — https://docs.litellm.ai/docs/
Statsig · LiteLLM cost tracking — https://www.statsig.com/perspectives/litellm-cost-tracking
StackPulsar · LiteLLM production monitoring (2026) — https://stackpulsar.com/blog/litellm-production-monitoring/

Energy per token in Spain: methodology, the electricity market and the cases of speculation

Sun, 14 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The whole article is focused on the Spanish electricity market (OMIE). The dollar symbol is not used (on this site it is a formula delimiter).

What this article covers

This is the second article of the energy track (C2), and deliberately longer, because it puts together two things that are almost never told together: the technical methodology of measuring energy per token, and the reality of the Spanish electricity market that puts a price on that energy. For an inference platform in Spain, the J/token figure is half the story; the other half is that those kilowatt-hours are bought in a market that is marginal-priced, volatile and with documented cases of manipulation. Anyone who sizes the electricity cost of a cluster without understanding how the price is formed in OMIE, and why it has tripled in a year, is building 30–50 % of the TCO on sand. This article covers the measurement, the market, the cases of speculation and the strategies for not being left exposed. No partisan opinions: data, market mechanics and documented facts, with their sources.

The identity: energy per token, and why the price rules

Energy per token comes from the identity set out in the opening article:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

And the electricity cost per token is that energy multiplied by the price of electricity:

$$\text{electricity cost per token} = \text{energy per token (kWh)} \times \text{PUE} \times \text{price (€/kWh)}$$

Here is the point of this article: the first two factors (energy and PUE) are controlled by your engineering; the third, the price, is set by the Spanish electricity market, and it is the one that moves most. The same workload, with the same efficiency, can cost three times as much in electricity depending on the month, not because of the technology but because of OMIE. That is why measuring energy per token well is necessary but not sufficient: you have to understand the price variable that multiplies it.

Methodology: measuring energy per token without fooling yourself

Before the market, the measurement. As we saw in the energy introduction, the energy-per-token figure has four instrumentation decisions that change the result:

Decision	Trap	Good practice
Sampling frequency	coarse sampling misses the prefill peaks	sample finely (sub-second)
Board vs node	the GPU alone underestimates by 10–20 %	measure the whole node or declare the limit
Time window	including warm-up or shutdown distorts the J/token	align the window with the measured workload
Idle baseline	attributing or subtracting changes the number	decide and document it

The golden rule: the same window for the power and for the tokens. If DCGM integrates the power of one interval and the benchmark tool counts the tokens of another, the J/token means nothing. With that settled, you already have the energy per token; the rest of the article is about what multiplies it.

The Spanish electricity market: how the price is formed

In Spain (and Portugal), the wholesale price of electricity is set by OMIE (Operador del Mercado Ibérico de Energía), which runs the day-ahead market and sets the price for each hour (OMIE). The system is marginal-priced (pay-as-clear): each hour supply (the plants, ordered from cheapest to most expensive) and demand are matched, and the price for everyone is that of the last plant to enter, the marginal one. That marginal price is the same across all of Spain (except the Canaries) and applies to all agents regardless of their initial bid (Energía y Sociedad).

The consequence, central to understanding the bill: when gas is the marginal plant (the usual case in hours without sun or wind), renewables, nuclear and hydro, dirt cheap to produce, are paid the price of gas. That difference between production cost and price received is what are called “windfall profits”, the heart of the debate about market design. For the buyer of electricity, a datacenter for example, it means that the price paid does not reflect the cost of generating, but that of the most expensive technology needed that hour.

The volatility: the price that triples in a year

The marginal-priced market, tied to gas in the expensive hours, produces enormous volatility. The 2026 data illustrates it: the Spanish wholesale price closed May 2026 with an average of 54 €/MWh, more than triple the 16.92 €/MWh of the same month in 2025 (Merca2). Tripling in a year, without your consumption or your efficiency changing.

And within a single day, the dispersion is brutal because of the effect of renewables: solar and wind enter at prices close to zero, and in the midday hours of May 2026 prices reached negative levels in several sessions (Merca2). The structural context: wind and solar went from 26 % of generation in 2019 to more than 40 % in the first half of 2024, and over that period the wholesale price was more than 40 % lower than it would have been with 2019 renewable levels (Banco de España).

Reference	Value	Source
Wholesale May 2026	54 €/MWh	Merca2
Wholesale May 2025	16.92 €/MWh	Merca2
Year-on-year variation	+219 % (×3.2)	derived
Solar hours 2026	prices close to 0 or negative	Merca2
Renewables in generation	26 % (2019) → >40 % (2024)	Banco de España

The reading for a cluster: the electricity part of the TCO is not a fixed number, it is a variable with a range of 3× year on year and from zero-price hours to peak hours within the day. Sizing the energy cost with an annual average price and forgetting about it is ignoring the biggest risk in the model.

The structure of the bill: wholesale is not what you pay

It is worth not confusing the wholesale price (the OMIE pool price, the one the headlines talk about) with what a company actually pays. On top of the wholesale price come:

Component	What it is	Nature
Energy (wholesale)	the OMIE price hour by hour	variable, volatile
Access tolls	use of the transmission and distribution networks	regulated
Charges	system costs (past renewables, etc.)	regulated
Taxes	IEE (special electricity tax) + VAT	fiscal

The result: the industrial retail price a datacenter pays is well above the wholesale price, since tolls, charges and taxes can double the price of pure energy. That is why this article always distinguishes between the wholesale price (~0.054 €/kWh in May 2026) and industrial retail (~0.12 €/kWh): for the TCO the second one matters, but the first is the one that moves and drags the second with it. A large company can access the wholesale market more directly (via a supplier or a representative), reducing the mark-up, but taking on the exposure to volatility that comes with it.

There is also the PVPC (Precio Voluntario al Pequeño Consumidor), the regulated tariff indexed to the wholesale price for small consumers; a datacenter does not use PVPC, but its existence explains why the volatility of the pool reaches public opinion and political debate directly, and, with it, the regulation that ends up affecting everyone.

The 2021–2022 crisis: the origin of the problem

Today’s volatility is not new: it originates in the gas crisis of 2021–2022. When the price of gas shot up (the marginal plant in many hours), the marginal-pricing system passed that price on to all electricity, including that generated by cheap renewables, nuclear and hydro. The result was wholesale prices that were several times the historical levels and massive “windfall profits” for generators of cheap technologies that were paid the price of gas. That episode is what motivated both the Iberian exception (capping gas in price formation) and the European debate about redesigning the market, and it explains why, years later, the price is still a first-order risk variable for anyone consuming a lot of megawatt-hours. For a multi-year TCO model, the lesson is that the Spanish electricity price has shown it can multiply within months through external factors (gas geopolitics), and that no assumption of a “stable” price survives an episode like that unless it is covered by contract.

The cases of speculation and manipulation

The volatility of marginal pricing creates incentives to manipulate the price, and in Spain there are documented cases with final CNMC fines. The largest sanctions on utilities for manipulating the wholesale market (Nada es Gratis):

Company	CNMC fine	Concept
Iberdrola	25 million €	manipulation of the wholesale price via hydro plants
Naturgy	19.5 million €	manipulation of the wholesale market
Endesa	5.8 million €	manipulation of the wholesale market

The Iberdrola case is the most illustrative of how marginal pricing is gamed: between 30 November and 23 December 2013, the company raised the prices of several hydro plants so that they would not clear despite the high prices in the day-ahead market (Nada es Gratis). The mechanism: withhold cheap water (not bidding it, or bidding it at a prohibitive price) so that the clearing needs a more expensive plant, pushing up the marginal price paid to your whole portfolio. In a pay-as-clear system, withdrawing cheap capacity raises the price of everything you sell that hour; the incentive to manipulate is embedded in the design.

OMIE has supervision mechanisms and carries out audits to prevent manipulation (Energía y Sociedad), and the CNMC sanctions it when it is detected; the fines above are the proof that it happens. For the buyer, the relevant point is not to assign blame, but to recognise that the price paid is formed in a market where manipulation is possible and has happened, and that therefore exposure to that price is a risk to manage, not a figure to accept.

The other speculation: hoarding grid capacity (and the new 2026 law)

There is a second form of speculation, distinct from price manipulation, that affects directly anyone who wants to build a datacenter: the hoarding of access and connection permits to the grid. For years, many developers reserved connection capacity at grid nodes for projects (mostly renewable) that had no real intention or capacity to build, blocking that capacity for firm projects and speculating on the value of the permit. Since an inference platform is a demand facility that needs its connection point, this bottleneck hits it squarely: grid capacity is a scarce and hoarded resource.

The regulatory response of 2026 targets exactly that. The Government (Ministry for the Ecological Transition) finalised a set of rules, within the framework of Royal Decree-Law 7/2026, to free up grid capacity and fight speculation, ensuring that firm requests can connect (El Periódico de la Energía, Fieldfisher). The central mechanism, with effect from 22 March 2026:

Element	Detail
Who pays	holders of access and connection permits for demand facilities
What they pay	a monthly charge for “reserving” capacity until entering operation
How it is calculated	transmission and distribution tolls × a “k factor”
Modulation	by voltage level and by the delay in commissioning
k factor	to be set by resolution of the Secretary of State for Energy

The logic: if reserving capacity costs every month and the cost grows with the delay, hoarding stops being free and the speculator releases the capacity it is not going to use. For a real datacenter project the implication is twofold: (1) that charge on the reserved capacity has to be budgeted during construction, and (2) there is a strong incentive to build quickly, since every month of delay in starting up adds to the charge. In other words, the new law turns commissioning time into an explicit cost. For the TCO model and the timetable of the proposal, this is a figure to build in from the start: grid capacity is no longer reserved “just in case” for free, and the construction schedule now has a bill attached.

The design debate: why marginal pricing rewards speculation

The underlying problem is not just that some companies manipulate: it is that the market design rewards doing it. In a pay-as-clear system (everyone is paid the marginal price), a generator with a lot of cheap capacity has a perverse incentive: withdrawing part of its cheap capacity raises the marginal price it is paid for all the capacity it does sell. If you control enough market share, the extra revenue from raising the price of your whole portfolio can exceed what you lose by not selling the withdrawn capacity. That is exactly what the CNMC sanctioned in the hydro case.

The design alternatives under debate:

Design	How it pays	Effect on the incentive to speculate
Pay-as-clear (current)	everyone is paid the marginal price	high: withdrawing capacity raises the price of everything
Pay-as-bid	each one is paid its own bid	lower incentive, but strategic bidding
Long-term contracts / PPA	fixed bilateral price	takes volume out of the pool, reduces exposure

There is no technical consensus that pay-as-bid is better (it introduces its own problems of strategic bidding), and the reform of the European electricity market has opted more for promoting long-term contracts (PPA and CfD) than for changing the pool mechanism. For the buyer, the practical conclusion is the same as for the regulator: taking your volume out of the pool through forward contracts is the most direct way of not being exposed to the marginal price, and to whoever manipulates it.

The Iberian exception: a patch with an expiry date

Faced with the gas price crisis of 2021–2022, the European Commission granted Spain and Portugal an exception allowing them to cap the price of the gas used to produce electricity (market context). The “Iberian exception” partially decoupled the electricity price from gas, containing the peaks. It is relevant for a multi-year cost projection because it is a regulatory mechanism with limited duration: any 36-month TCO model has to consider what happens to the price if the exception changes or expires. Regulation is another price variable, not just the market.

What it means for the electricity cost of a cluster

Putting together energy per token (technical) with the Spanish price (market), the electricity cost per token of an example node (8×H100, 0.78 kWh per million tokens with PUE) according to the price scenario:

Price scenario (Spain)	€/kWh	Electricity cost / 1M tokens
Solar hour / negative price	~0	~0
May 2025 (low wholesale)	0.017	~0.013 €
May 2026 (average wholesale)	0.054	~0.042 €
Industrial retail (with tolls and taxes)	~0.12	~0.094 €
Peak hour / crisis	0.20+	~0.16 €

The range is more than 10× between the solar hour and the peak hour, and 3× year on year at the level of the average. Over a fleet consuming megawatt-hours a year, that variability is the difference between a comfortable TCO and a suffocated one, and it does not depend on your technology but on when and how you buy the energy.

Strategies for not being left exposed

This is where the technical figure turns into a decision. Four levers for managing exposure to the Spanish price, from the least to the most structural:

Strategy	What it does	Trade-off
Indexed tariff	you pay the wholesale price hour by hour	cheap on average, maximum exposure to volatility
Fixed / forward tariff (OMIP)	price locked by contract	predictability, you pay a premium for the insurance
PPA with a renewable plant	you buy directly from a solar/wind farm	stable price and low carbon, requires volume and term
Solar self-consumption	you generate on site	you cover the sunlight hours almost free, capex and space
Temporal scheduling	move shiftable load to cheap hours	only applies to the non-urgent (batch, fine-tuning)

The sensible combination for an inference platform in Spain: a PPA or a fixed tariff for the base load (online inference, which cannot be moved), solar self-consumption to cover the midday hours almost free, and scheduling of the shiftable work (training, overnight batch) towards the off-peak or peak-solar hours. That turns the volatility of marginal pricing from risk into opportunity: the same zero-price hours that sink generators’ revenue are the cheap hours for whoever consumes.

Solar self-consumption: the numbers for a cluster

Spain is one of the best places in Europe for solar, and for a datacenter self-consumption has a clear cost logic. The numbers, for illustration:

Parameter	Reference value
Solar output in Spain	~1,400–1,600 kWh per kWp per year
Capex of an industrial installation	~600–900 € per kWp
Panel service life	~25–30 years
Electricity cost avoided (retail)	~0.12 €/kWh

For a 100 kWp installation (warehouse roof): it generates ~150 MWh/year, with a capex of ~70,000–90,000 €. At 0.12 €/kWh avoided, it saves ~18,000 €/year, with a payback of ~4–6 years and then ~20 years of almost free energy. The snag: a cluster consumes 24/7 and solar only produces by day, so self-consumption covers a fraction, the midday hours, not everything. But those midday hours are precisely the ones with low or negative wholesale prices, so self-consumption and the market complement each other: at night you buy cheap (off-peak), by day you generate. The avoided cost is not only that of the energy, it is that of not being exposed to the non-existent solar peak and to daytime volatility. For a cluster with roof space available, self-consumption is the most solid structural lever on electricity cost in Spain.

Annual electricity TCO: three scenarios for a cluster in Spain

Putting it all together, the annual electricity cost of an 8×H100 node (~7.84 kW with PUE, ~68,700 kWh/year at 100 % utilisation) according to the purchasing strategy, in Spain:

Strategy	Effective €/kWh	Annual electricity cost	Exposure
Indexed (unhedged)	~0.08 (variable)	~5,500 € ±a lot	maximum to volatility
Fixed / renewable PPA	~0.06 (stable)	~4,120 €	low, predictable
PPA + solar self-consumption	~0.045 (mixed)	~3,090 €	low + minimal carbon

The difference between the worst and the best strategy is ~2,400 €/year per node, before counting the risk: the indexed option can shoot up in a bad month, while PPA+solar is stable and low. Over a fleet of dozens of nodes and a 36-month horizon, the choice of how the energy is bought moves the TCO as much as a hardware decision, and it is a lever decided with contracts, not with engineering. This number feeds directly into the TCO model of the FinOps track (A8) and the sizing of the proposal.

PPAs: the central lever (and its small print in 2026)

Of all the strategies, the PPA (Power Purchase Agreement, a long-term energy purchase contract) is the most important piece for a datacenter in Spain today, and it deserves a section of its own because its current situation is exceptionally favourable… with a catch.

What a PPA is, in short

A PPA is a bilateral, long-term contract (typically 10–12 years in Spain) between a consumer and a renewable generator, at a locked price per MWh. It takes your volume out of the OMIE pool: instead of paying the marginal price hour by hour (with its volatility and its exposure to manipulation), you pay the contract price. There are two families: physical (you receive the energy from that farm) and financial/virtual (a swap covering the difference against the market); and two profile forms: pay-as-produced (you pay for what the farm generates, when it generates, cheap but intermittent) and baseload (a guaranteed flat profile, more expensive but firm).

Prices today: historic lows

Spain is living a unique moment in PPA prices. Solar PPAs fell to a historic low of 32.5 €/MWh in the fourth quarter of 2025, with the most competitive percentiles below 30 €/MWh, half the French level (pv-magazine, El Periódico de la Energía). Wind sits at around 50–60 €/MWh. To put it in context: a solar PPA at 30 €/MWh is ~0.030 €/kWh, less than half the industrial retail price (~0.12 €/kWh) and well below the volatile wholesale price.

Technology (PPA, Spain 2026)	Price	Reference
Solar photovoltaic	~32.5 €/MWh (percentiles <30)	pv-magazine
Wind	~50–60 €/MWh	market
Industrial retail (for comparison)	~120 €/MWh	—

Why so cheap: cannibalisation

The fall is no accident: it is the cannibalisation of photovoltaics. The more solar is installed, the more midday hours there are with wholesale prices close to zero or negative, which sinks the value captured by solar (captured prices have fallen below 15 €/MWh), and with it the price at which farms sign PPAs (Energía Estratégica). With more than 40 GW competing for 30 GW of demand, there is an excess of solar supply that cheapens the PPA. For the industrial consumer it is a historic opportunity; for the solar developer, a crisis of profitability, hence the talk of the “closing of the solar PPA window” and of the new Royal Decree seeking to restore industrial competitiveness.

Spain, European leader in PPAs

This is no marginal phenomenon: in 2024 Spain was the country with the most PPAs signed in Europe for the sixth consecutive year, with 47 agreements for 4.66 GW of renewable capacity (market sources). The market is deep, liquid and mature, which makes it easier for a datacenter to find a counterparty.

The catch for a datacenter: annual energy ≠ firm power

Here is the small print that decides the architecture. A cheap solar PPA offsets energy over the year, but does not guarantee physical power every hour. And an AI datacenter cannot stop because there is no wind one January night or because the solar curve falls at dusk (El Periódico de la Energía). This is the difference between energy (kWh per year, which the solar PPA covers dirt cheap) and firm power (kW available for all 8,760 hours, which the sun does not provide). The sector sums it up like this: Spain is now forced to talk about firm energy (Revista Cloud).

The consequence for the design of the proposal: a solar PPA at 30 €/MWh is the ideal cost floor for the sunlight hours, but it has to be combined to cover the rest: a hybrid PPA (solar + wind, which complement each other over time), baseload for the flat part, or storage (batteries, which RDL 7/2026 also promotes) to move energy from the sunlight hours to the night. The sensible energy architecture of a Spanish datacenter in 2026 is not “a solar PPA”, it is a cheap solar PPA as the daytime base + firmness for the hours without sun (wind, baseload, batteries or the grid as backup). Anyone who signs only the cheap solar PPA and discovers in production that they have no firm power in the small hours will have optimised the cost of the energy and ignored that of availability, which for an inference platform with an SLO is non-negotiable.

The context: the datacenter boom in Spain

All of this is happening in the middle of a boom: Spain is positioning itself as the epicentre of southern Europe in datacenters, with investments of billions and installed capacity doubling towards 2026 (DCD). The appeal combines cheap renewable PPAs, industrial land more accessible than in other European digital capitals, and EU jurisdiction. The constraint that tempers it is exactly the one in C2: the cost of energy is low but firm power and grid capacity are the bottlenecks, and both are managed with the combination of a hybrid PPA, storage and a connection schedule that does not fall foul of the hoarding charge.

The sovereign and carbon angle

For a sovereign proposal, Spain has a double advantage worth quantifying. First, carbon: the Spanish grid runs at around ~150–170 gCO₂/kWh (a mix of renewables and gas), well below Germany (~363) though above nuclear France (~20–60). Second, growing renewables (>40 % of generation), which pull the price and the carbon down in the sunlight hours. The argument for the proposal: a cluster in Spain, with solar self-consumption and a renewable PPA, can achieve at the same time a low and predictable electricity cost and a competitive carbon per token, within EU jurisdiction. The volatility of marginal pricing is not only a risk: managed well with contracting and scheduling, it is a cost lever that a hyperscaler in another region does not exploit in the same way.

Carbon, moreover, is variable by the hour: in peak-solar hours the intensity of the Spanish grid drops, so moving shiftable load to those hours reduces both the cost and the carbon per token at once. The same scheduling strategy serves both axes.

Guarantees of Origin: carbon has its market too

One nuance that connects price speculation with carbon reporting: in Spain (and the EU) there is a market in Guarantees of Origin (GoO), certificates attesting that a quantity of energy was produced from a renewable source. A company can buy GoOs to declare that its consumption is “green” in its market-based carbon reporting, even though physically the electricity it received from the grid had the average intensity of the mix. This creates two ways of accounting for carbon, with different implications:

Method	What it counts	How the figure is brought down
By location (location-based)	the real intensity of the Spanish grid that hour	generate/consume in clean hours; location
By market (market-based)	according to the GoOs/PPAs contracted	buy certificates or sign a PPA

Location-based accounting rewards consuming when the Spanish grid is clean (solar hours); market-based allows the green attribute to be “bought” with GoOs. CSRD reporting normally requires both. The word of caution: the GoO market is relatively cheap, and buying GoOs without a real PPA behind them is a way of dressing up the figure without changing the physical electricity you consume, the cost equivalent of accounting greenwashing. For an honest sovereign proposal, the robust route is a real PPA with a Spanish renewable farm (which does change the physical origin and stabilises the price) plus location-based measurement that rewards scheduling to clean hours, rather than accumulating certificates.

Hourly intensity: the Red Eléctrica figure

Spain publishes the carbon intensity of the grid hour by hour (via Red Eléctrica / esios and ElectricityMaps), and it varies enormously over the day: on a sunny spring afternoon it can drop to ~80–100 gCO₂/kWh, and on a cold, windless night, with gas cycles, exceed ~250. That hourly figure is what makes carbon-aware scheduling possible: moving training, document ingestion or overnight batch to the hours of lowest intensity reduces the carbon per token of the shiftable work at no technical cost. It is the same lever as price-based scheduling, since the cheap hours and the clean ones largely coincide in Spain, because both are set by the excess of solar and wind. Measuring carbon per token “on an annual average” wastes this information; measuring it hour by hour and scheduling accordingly is what turns the figure into a real reduction.

Energy per token, with the Spanish price, step by step

Closing the circle with a complete calculation for the example node, in Spain:

Energy per token: 5,600 W (8×H100) ÷ 2,800 tok/s = 2 J/token (board); ×PUE 1.4 = 2.8 J/token effective.
Energy per 1M tokens: 0.78 kWh (with PUE).
Electricity cost per 1M tokens by contracting model:
- Indexed in a solar hour (~0 €/kWh): ~0 €.
- Renewable PPA (~0.05 €/kWh stable): ~0.039 €.
- Industrial retail (~0.12 €/kWh): ~0.094 €.
Carbon per 1M tokens: 0.78 kWh × ~160 gCO₂/kWh = ~125 gCO₂ (and less in solar hours).

The message: the J/token is fixed (your engineering sets it); the electricity cost per token varies more than 10× depending on how and when you buy the energy in Spain. That is why this article devotes more space to the market than to the measurement: measuring is 30 % of the problem; buying well is 70 %.

Limits and traps (data-driven)

A misleading annual average price. Using an annual average hides the 3× year-on-year volatility and the 10× intraday range. Model scenarios, not a point.
Indexed by default. The indexed tariff is cheap on average but leaves you exposed to every peak and every episode of manipulation. For base load, hedge it.
Ignoring the Iberian exception. It is a regulatory mechanism with an expiry date; a 36-month TCO has to consider it changing.
Forgetting hourly carbon. Carbon and price fall at the same time in solar hours; scheduling the shiftable work improves both axes.
Confusing wholesale with what you pay. Industrial retail adds tolls and taxes on top of the wholesale price; use your contract price, not the pool price.

With the Spanish price understood, the energy track continues towards measurement tools in depth (C3) and energy in TCO and regulation (C8). But the lesson of C2 is that, in Spain, the energy cost of AI is not decided in the GPU alone: it is decided in how you buy the electricity in a market that is marginal-priced, volatile and watched for manipulation.

Closing

Energy per token is a number that engineering fixes with precision, so many joules per token, measured in a window aligned with the throughput, and that the Spanish electricity market multiplies by a factor that changes more than 10× depending on the hour and 3× depending on the year. That second half, the price one, hardly ever appears in technical AI articles, and it is what decides 30–50 % of the TCO of a cluster in Spain. The price is formed in a marginal-priced market where cheap renewables are paid the price of gas, where volatility has tripled the bill in a year, and where there are final CNMC fines, Iberdrola 25 M€, Naturgy 19.5 M€, Endesa 5.8 M€, that show that manipulation is not theoretical. For a sovereign inference platform in Spain, the conclusion the data supports is twofold: measure energy per token rigorously, and buy the electricity with your head — a PPA or fixed tariff for the base load, solar self-consumption for the sunlight hours, and scheduling of the shiftable work to the cheap, clean hours. Spain offers abundant sun, growing renewables and a reasonable average carbon figure; well contracted, its energy is a competitive advantage, and badly contracted, the biggest risk in the model. The J/token is yours to decide; the euro per token, the market’s, unless you take it out of the market with a contract.

Sources

OMIE · Operador del Mercado Ibérico de Energía — https://www.omie.es/
OMIE · precio del mercado diario — https://www.omie.es/es/market-results/daily/daily-market/day-ahead-price
Energía y Sociedad · importancia y precios de OMIE — https://www.energiaysociedad.es/omie-cual-es-su-importancia-para-el-sistema-electrico-espanol-y-portugues/
Nada es Gratis · manipulación del precio en el mercado eléctrico español (multas CNMC) — https://nadaesgratis.es/jose-luis-ferreira/manipulacion-del-precio-en-el-mercado-electrico-espanol
Merca2 · el precio de la luz en mayo 2026 (54 €/MWh, ×3 interanual) — https://www.merca2.es/2026/05/30/precio-luz-mayo-2026-disparo-2386473/
Banco de España · el sector eléctrico ante el alza del gas (renovables y precio) — https://www.bde.es/f/webbe/SES/Secciones/Publicaciones/PublicacionesSeriadas/DocumentosOcasionales/23/Fich/do2316.pdf
El Periódico de la Energía · normativa para liberar capacidad de red y combatir la especulación — https://elperiodicodelaenergia.com/el-gobierno-ultima-una-normativa-para-liberar-la-capacidad-de-las-redes-electricas-y-combatir-la-especulacion/
Fieldfisher · Real Decreto-ley 7/2026, paquete regulatorio de renovables — https://www.fieldfisher.com/es-es/locations/espana/actualidad/nuevo-paquete-regulatorio-energias-renovables
Bird & Bird · almacenamiento energético en el RDL 7/2026 — https://www.twobirds.com/es/insights/2026/spain/almacenamiento-energetico-real-decreto-ley-7-2026
pv-magazine · caída récord de los PPAs fotovoltaicos a 32,5 €/MWh — https://www.pv-magazine.es/2026/02/04/la-canibalizacion-provoca-una-caida-record-de-los-precios-de-los-ppas-fotovoltaicos-en-espana-hasta-los-325-e-mwh/
El Periódico de la Energía · PPAs solares cerca de 30 €/MWh (mitad que Francia) — https://elperiodicodelaenergia.com/la-canibalizacion-acerca-el-precio-de-los-ppas-solares-a-30-mwh-en-espana-la-mitad-que-en-francia/
Energía Estratégica · cierre de la ventana de PPAs solares y el nuevo RD — https://www.energiaestrategica.com/espana-atraviesa-un-cierre-de-la-ventana-de-ppas-solares-mientras-el-nuevo-real-decreto-busca-aliviar-la-industria/
El Periódico de la Energía · la gran decisión energética de los data centers (energía firme) — https://elperiodicodelaenergia.com/la-gran-decision-energetica-de-los-data-centers
Revista Cloud · los centros de datos obligan a España a hablar de energía firme — https://revistacloud.com/los-centros-de-datos-obligan-a-espana-a-hablar-de-energia-firme
DCD · boom de data centers en España hacia 2026 — https://www.datacenterdynamics.com/es/features/data-centers-en-espa%C3%B1a-hacia-2026-un-boom-de-90000-millones-en-proyectos/

OpenCost in depth: how GPU cost is allocated in Kubernetes

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. European references (OVH is among the supported providers). The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Second article in the FinOps track (A2), and the first tool deep dive. The FinOps introduction established that OpenCost is the CNCF standard for cost allocation; here the box gets opened: how it allocates cost internally, where it gets its data from, how on-prem prices are configured in euros, what its API exposes, and where the traps are. Understanding the mechanics matters because it defines what any tool on top (Kubecost included) can and cannot do, and because a badly set base price invalidates the whole split. No recommendations; only the mechanics and the methodology.

What OpenCost is

OpenCost is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project). It is a cost allocation layer for Kubernetes: it reads resource usage from the cluster, joins it with a price, and splits the cost across Kubernetes dimensions down to the container (OpenCost · GitHub). It supports dynamic pricing via the billing APIs of AWS, Azure and GCP, and custom prices for on-prem clusters (OpenCost · configuration).

What it is not: it is not an optimiser, nor a governance system, nor a product unit economics tool. It is the piece that answers “how much does each namespace/pod/team cost?”, and leaves the rest to the layers above. That is why it is the foundation of the FinOps track: without correct allocation, there is no cost per token and no TCO model worth anything.

The cost model: node-level allocation

The key to OpenCost is that the model works at node level. It starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit per-resource prices, OpenCost uses the ratio of a set of base prices (marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem).

This is what makes it work on-prem: you declare the node cost (depreciated capex + opex), and OpenCost splits it without needing a cloud invoice. The normalisation guarantees that no cost is “lost” or “invented”: what you pay for the node is exactly what gets split across its resources.

Where the data comes from: Prometheus

OpenCost instruments nothing of its own: it reads from Prometheus, which is a prerequisite of the installation (OpenCost · GitHub). The sources:

Source	What it provides
`kube-state-metrics`	state of K8s objects (pods, requests/limits)
`node-exporter`	node resources and capacity
`cAdvisor`	real CPU/memory usage per container
`dcgm-exporter` (NVIDIA)	GPU usage, memory and power

With those series, OpenCost automates the join between Kubernetes resource usage and the provider price, reducing the custom PromQL you would otherwise write by hand (Grafana · manage costs). The GPU signal comes from DCGM via the dcgm-exporter (part of the NVIDIA GPU Operator), the same base as GPU observability. Without Prometheus and without DCGM exporting, OpenCost does not see the GPU.

Architecture: the pieces

OpenCost is not a monolith; it is three responsibilities worth telling apart:

Piece	Function
Cost model	resolves the node price and splits/normalises it across resources
Allocation API	serves the allocated cost per dimension (`/allocation` and endpoints)
Exporter	publishes cost metrics on `/metrics` for Prometheus/Grafana

The data flow: Prometheus holds the usage series (from kube-state-metrics, cAdvisor, dcgm-exporter); the cost model joins them with the resolved price; the Allocation API serves them aggregated by whatever dimension you ask for; and the exporter returns them to Prometheus so that Grafana can plot them. It is a cycle: metrics come in from Prometheus and cost goes back to Prometheus enriched. That symmetry is what lets you reuse your observability stack without standing up a new database: OpenCost lives on top of Prometheus, not beside it.

An operational note: historical analysis depends on Prometheus retention. For cost reports spanning months, a long-term backend behind it (Thanos, Mimir, VictoriaMetrics) is advisable, or periodically exporting the allocations to a store of your own.

On-prem prices in euros: the piece that decides everything

On-prem, the most important figure, and the most neglected, is the node price. It is configured per provider; the options are alibaba, aws, azure, gcp, oracle, ovh or default (on-prem) (OpenCost · configuration). For on-prem you use default, supplying base prices in a default.json (or by overriding the Helm values.yaml); anything you do not override uses the chart’s value (OpenCost · on-prem).

To set the node price in euros, the calculation is the TCO model one: depreciated capex + opex. An example of an 8×H100 node:

Component	Calculation	€/hour
Capex (node ~240,000 €, 36 months)	240,000 ÷ 26,280 h	~9.13
Energy (5.6 kW × PUE 1.4 × 0.058 €/kWh, France)	~0.455 €/h	~0.46
Maintenance, networking, operations	estimate	~1.5
Node total		~11.1 €/h

That ~11 €/h is the number you put into OpenCost, and everything is split from it. And here is the documented trap: with the default pricing configuration, OpenCost under-prices CPU and GPU on-prem (issue #3781). If you do not adjust the base prices to your real cost, the split will be systematically low and the cost per token you report will be unreal. Configuring the node price is not optional: it is the figure everything else hangs from.

Configuring the price: an example

The base price is given in the Helm values.yaml (or a default.json), in euros per hour and per unit of resource, derived from the node’s total cost:

# values.yaml — on-prem prices in €/hour (8xH100 node, France)
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 8xH100 node"
 CPU: "0.030" # €/CPU-hour
 RAM: "0.004" # €/GB-hour
 GPU: "1.30" # €/GPU-hour ← the figure that moves cost/token the most
 storage: "0.0002" # €/GB-hour

OpenCost normalises those values so that, multiplied by the node capacity, they add up to the total price you declared (~11 €/h in the example). The number to be most careful with is the GPU one: at 1.30 €/GPU-hour, the 8 cards are ~10.4 €/h, the bulk of the node’s cost. A badly set GPU base shifts the whole cost per token. Always review the resolved prices in /allNodePricing after configuring, because the default under-pricing bug bites exactly here.

Allocation: by usage, not by presence

The behaviour that has changed most and that matters most for GPU: in recent versions, the cost allocated to a GPU is determined by its usage, not its presence. If a GPU costing 100 €/month is only half used, the allocated cost is ~50 € ([search]). This aligns allocation with reality, you pay for what you use, but it has a consequence: the other half does not disappear, it is idle that somebody keeps paying for. OpenCost exposes it, and that is where the optimisation lever comes from.

OpenCost allocates to any Kubernetes dimension: cluster, node, namespace, controller (deployment, statefulset and so on), service, pod and container, and by labels (team, product). Access is through the Allocation API and several endpoints that expose the mechanics:

Endpoint	What it returns
`/allocation`	cost allocated by the dimension you ask for
`/costDataModel`	the resolved node price
`/allNodePricing`	hourly prices per node
`/pricingSourceSummary`	summary of the pricing source
`/metrics`	cost metrics for Prometheus

The /metrics endpoint is what turns OpenCost into a Prometheus exporter: once there, you can write PromQL to calculate the cost and efficiency of any Kubernetes concept and build panels in Grafana (OpenCost · exporter).

Allocation API parameters

/allocation is controlled by a handful of parameters worth knowing, because they define which number you get:

Parameter	What it controls	Example
`window`	time range	`7d`, `today`, specific dates
`aggregate`	aggregation dimension	`namespace`, `label:team`, `pod`
`accumulate`	sum the range or break it per interval	`true` (total) / `false` (series)
`idle`	include idle cost or not	`true` / `false` / `separate`
`filter`	filter by namespace, label and so on	`namespace:"llm-prod"`

The idle parameter is the most revealing: with idle=separate, OpenCost returns the idle cost as a separate row, which lets you see at a glance how much is being paid for unused capacity. A query with aggregate=label:team and idle=separate over window=30d is, literally, the monthly chargeback report with the waste highlighted.

GPU in depth: DCGM, usage, MIG and idle

The GPU is the expensive resource, so its allocation deserves detail:

Usage signal: dcgm-exporter gives DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (memory), and power. OpenCost uses utilisation to allocate by usage.
MIG (Multi-Instance GPU): an A100 is split into up to 7 isolated instances, which allows cutting idle from 50 % to almost 0 % by sharing one card across small workloads ([search]; see sharing the GPU: time-slicing, MPS and MIG). The cost allocation of each MIG instance follows the same usage logic.
Idle: the reliable detection pattern combines three pieces, DCGM metrics in Prometheus, an alert that fires when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, and routing of that alert to the team that owns the namespace ([search]).

An attribution nuance worth knowing: MIG creates isolated instances that Kubernetes sees as distinct resources, so OpenCost allocates them cleanly, each one to its pod. Time-slicing, by contrast, shares the same physical GPU across several pods with no compute isolation: there, per-pod utilisation is harder to separate, and cost attribution becomes approximate (the GPU is split across the pods sharing it, according to their measured usage). The practical rule: if you need exact per-team chargeback on a shared GPU, MIG gives cleaner attribution than time-slicing, at the price of rigid fixed partitions. It is a trade-off between cost accuracy and flexibility that is better decided before, not after, building the split.

Efficiency: the number that triggers action

With cost allocated by usage, OpenCost lets you calculate efficiency: how much of the allocated cost corresponds to useful work against the total. A GPU at 30 % utilisation has an efficiency of 30 %: the remaining 70 % is idle cost that somebody pays without receiving work. That figure, per team and per namespace, is what turns a dashboard into a chargeback conversation: “your namespace has 5 GPUs allocated at 25 % efficiency; either you raise utilisation or you release three cards”. It connects with FinOps and multi-tenancy: allocation locates the waste; scheduling and co-residency recover it.

Worked example: the allocation report

On the ~11 €/h node (8×H100), three teams share the cluster for a month. What a /allocation?aggregate=label:team&idle=separate&window=30d query returns:

Team	GPU-hours	Avg. util.	Allocated cost (€)	Efficiency
A · chat-prod	2,880 (4 GPU)	78 %	~3,744	high
B · batch	1,440 (2 GPU)	55 %	~1,872	medium
C · experimentation	1,440 (2 GPU)	22 %	~1,872	low
idle (separate)	—	—	~1,100	—

What the report reveals: the cost allocated to A, B and C comes out of their GPU-hours and the node price; but the idle row (~1,100 €/month) is capacity nobody used and everyone pays for in the pro rata. And the efficiency column points at C: 2 GPUs at 22 % is ~1,560 € a month of which only ~340 € is useful work. That is the actionable figure: C does not have a model problem, it has a utilisation problem. Without OpenCost, those ~1,100 € of idle and C’s inefficiency stay diluted in an aggregate invoice nobody questions. With it, they are a named row and a number in euros.

Useful PromQL queries

As a Prometheus exporter, OpenCost lets you build the panels by hand. Some reference queries (exact metric names vary by version; check in your /metrics):

# Cost per node and hour (resolved price)
node_total_hourly_cost

# GPU cost allocated per namespace (€/hour)
sum by (namespace) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
)

# Average GPU utilisation per namespace (for efficiency)
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

# Idle GPUs: utilisation < 10 % sustained
DCGM_FI_DEV_GPU_UTIL < 10

The last one, wrapped in an alert with for: 15m and routed to the namespace owner, is the standard idle detection pattern. With these series in Grafana you have cost per team, efficiency and waste on one dashboard, the “Inform report” of the FinOps phase in real time.

Dynamic cloud prices vs custom on-prem prices

The difference is worth understanding, because it changes how reliable the figure is:

Cloud (aws/azure/gcp/oracle/ovh): OpenCost obtains the price dynamically from the provider’s billing API, so the allocated cost reflects the real rate (including discounts, spot and so on) without you declaring anything.
On-prem (default/custom): you declare the price, derived from your TCO model. The advantage is total control; the risk is that a badly calculated price skews everything. That is why the TCO model (depreciated capex + opex) is not an academic exercise: it is literally OpenCost’s input.

The ovh provider among those supported is relevant for a European platform: if part of the workload goes to a European sovereign cloud, OpenCost can allocate its cost with dynamic pricing, and mix it with on-prem in the same allocation view.

From pod cost to cost per token

OpenCost gets as far as “this vLLM pod cost X €/hour”. The cost per token, the metric that compares on-prem against cloud, needs one more piece: the gateway (LiteLLM) that counts tokens per request and per team, as covered in the FinOps introduction. The full chain is: OpenCost gives the pod cost by usage → the gateway gives the tokens per team → dividing them gives the cost per token per team. OpenCost is the hardware half of that equation; without it, you would know the tokens but not their real cost. That is why A2 (allocation) is the foundation the whole cost per token of the proposal hangs from.

Deployment: Helm + Prometheus

The typical deployment is via Helm, with Prometheus as a prerequisite (for scraping and series storage). Conceptual steps:

Have Prometheus with kube-state-metrics, node-exporter, cAdvisor and dcgm-exporter.
Install OpenCost with Helm, pointing it at your Prometheus.
Configure the node price in euros (default.json / values.yaml), the step most often neglected.
Expose /metrics and build panels in Grafana with PromQL over the cost metrics.

As a Prometheus exporter, OpenCost lets you write PromQL for the cost and efficiency of any concept and create bespoke dashboards (OpenCost · exporter). It reuses infrastructure a GPU cluster already has for observability (DCGM, Prometheus, Grafana): OpenCost only adds the pricing and allocation layer.

OpenCost vs Kubecost (just the essentials)

OpenCost is the free base; Kubecost (IBM) is the commercial product built on it. Briefly, what the commercial one adds:

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓
GPU via DCGM	✓	✓ (3.0)
Long historical retention	depends on your Prometheus	included
Automatic rightsizing	—	✓ (Turbonomic)
Governance, alerts, enterprise RBAC	basic	✓
Commercial support	community	✓ (IBM)

For most on-prem clusters with a platform team, OpenCost covers allocation; Kubecost adds value when you want automated optimisation and enterprise features without operating it yourself. The detail of the comparison, and when paying is worth it, is article A3.

Implementation checklist

For OpenCost to give a defensible per-team cost, order matters:

Step	Action	Verification
1	Prometheus with kube-state-metrics, node-exporter, cAdvisor	series present
2	NVIDIA GPU Operator + dcgm-exporter	`DCGM_FI_DEV_GPU_UTIL` in Prometheus
3	Install OpenCost (Helm) pointing at Prometheus	`/allocation` responds
4	Configure the node price in € (capex+opex)	`/allNodePricing` shows the real one
5	Label pods/namespaces by team and product	`aggregate=label:team` splits correctly
6	Grafana panels for cost, efficiency and idle	the waste is visible
7	Idle alert (`util<10` `for: 15m`) to the owner	the alert reaches the team
8	(optional) long-retention backend	months of history available

Step 4 is the one most often skipped and the one that most invalidates the result. If you only do one thing right, make it that one: without the correct node price in euros, steps 5 to 7 split a wrong cost with great precision.

Limits and traps (data-driven)

Default under-pricing. With the default pricing config, OpenCost underestimates on-prem CPU and GPU (#3781). Adjust the base prices to your real cost or the whole split will come out low.
It depends on Prometheus and DCGM. Without the right series (especially dcgm-exporter), there is no GPU allocation. Input data quality rules.
Allocation by usage ≠ total cost. A GPU at 50 % charges half; the other half is idle that still costs. Do not confuse “allocated cost” with “cost paid”.
Prometheus retention. Historical cost analysis depends on how much your Prometheus retains; for long series, plan a long-term backend.
Allocation, not per-token measurement. OpenCost reaches the pod; cost per token needs the gateway (LiteLLM) above it, as covered in the FinOps introduction.

With OpenCost’s mechanics clear, the next article (A3) compares what Kubecost and the commercial alternatives add, to decide which FinOps stack to adopt. But the foundation, correct allocation with the node price properly set in euros, is this one.

Closing

OpenCost looks like “installing a cost tool” and is, in reality, declaring what your hardware is worth and letting it split itself. That simplicity is its strength, it lives on top of the Prometheus and DCGM you already have, with no new database, and also its trap: the split is only as good as the node price you declare, and the default value under-prices on-prem GPU. Properly configured in euros, OpenCost turns the cluster’s aggregate invoice into a split per team, per product and per GPU, separates used cost from idle, and exposes the efficiency that triggers chargeback conversations. It is the hardware half of the cost per token equation, the other half, the tokens, is supplied by the gateway, and without this half there is no TCO model or on-prem vs cloud comparison that holds up. For a sovereign architecture proposal, OpenCost is the piece that makes cost stop being an intuition and become a table in euros, reproducible and auditable, with the idle GPU named and identified. The Inform phase of GPU FinOps starts here, and everything else hangs on this number being right.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · configuración (proveedores: aws/azure/gcp/oracle/ovh/default) — https://opencost.io/docs/configuration/
OpenCost · configuración on-prem (precios personalizados) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exportador de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
OpenCost · issue #3781 (infra-precio CPU/GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Grafana · manage costs (OpenCost + Kubernetes) — https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/

Cost, performance and energy: the three axes of LLM inference and the identity that ties them together

Sat, 13 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. Cost and energy references are European (France, Germany, Spain), because the aim is to defend a sovereign European architecture; when a source quotes prices in dollars it is flagged as “USD”. The dollar symbol is not used (on this site it is a formula delimiter).

What this series is

This is the first instalment of a series of 28 data articles (not opinion pieces) about the three dimensions that decide whether a sovereign European LLM platform is sustainable: cost (FinOps), performance (benchmarking) and energy. The aim is to have a body of verifiable figures, with their methodology, to support architecture decisions with numbers rather than intuition. This first article sets the frame: what each axis measures, how they link to one another, an end-to-end worked example with European data and the 2026 references that will be used as a base.

The quantitative thesis, in one sentence: the three axes are not independent. Throughput (tokens per second) is the common denominator of cost per token and energy per token, and energy is, on top of that, between 30 % and 50 % of the cost itself. Optimising one moves the other two. And in Europe, where the cluster sits (France, Germany, Spain) changes both the electricity cost and the carbon per token at once.

The three axes and their metrics

Axis	Question	Key metrics	Unit	Tool family
Cost (FinOps)	How much does each unit of work cost?	cost/1M tokens (CPM), cost/request, €/GPU-hour, utilisation	€/1M tok, €/h	OpenCost, Kubecost, CloudZero
Performance	How much work per second and at what latency?	TTFT, TPOT/ITL, throughput, goodput, P50/P99	tok/s, ms	GuideLLM, AIPerf, LLMPerf, MLPerf Inference
Energy	How many watts per unit of work?	energy/token, average power, PUE, carbon	J/tok, Wh, kWh, gCO₂	Zeus, CodeCarbon, MLPerf Power, Kepler

Each axis has its own in-depth introductory article in this same batch; here only how they fit together matters. It is worth breaking each one down a little more before crossing them.

Cost axis in detail

Metric	Definition	Why it matters
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	the comparable unit on-prem vs cloud
Cost per request	cost attributed to one complete request	billing by real usage
€/GPU-hour	hourly cost of a GPU (amortised or rented)	the basis of any calculation
Utilisation	fraction of the GPU actually used	the lever that moves everything else
Fixed vs variable cost	amortised capex vs energy/operations	defines the break-even point

Performance axis in detail

Metric	Definition	Dominant phase
TTFT	latency to the first token	prefill
TPOT / ITL	time between output tokens	decode
Throughput	tokens or requests per second of the system	both
Goodput	throughput that meets the SLO (not the raw figure)	the honest number
P95 / P99	latency percentiles (not the mean)	the real experience

Energy axis in detail

Metric	Definition	Unit
Average power	average instantaneous consumption	W
Energy per token	energy per generated token	J/token
Total energy	integral of power over time	Wh, kWh
PUE	datacenter overhead (cooling, losses)	ratio ≥ 1
Carbon	energy × grid intensity	gCO₂eq

The identity that ties the three axes together: throughput

The cost per million tokens (CPM) is the cluster’s cost divided by what it produces:

$$\text{CPM} = \frac{\text{cluster cost per hour}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

Energy per token has exactly the same form, with power in the numerator:

$$\text{energy per token (J)} = \frac{\text{average power (W)}}{\text{throughput (tok/s)}}$$

The two share the same denominator: throughput. Hence the key operational consequence: every performance improvement that raises tokens/s lowers both cost per token and energy per token at the same time. These are not three separate optimisations; throughput moves them together. That is why performance benchmarking (axis 2) is, at bottom, also a FinOps tool and an energy efficiency tool.

The CPM↔throughput relationship is direct: the higher the throughput, the lower the CPM. It is the identity on which the whole cost analysis of the series is built.

Worked example: unit economics of an 8×H100 node in Europe

To see the identity in action, an end-to-end calculation on an example node (8×H100 SXM, Llama 3.1 70B FP16 served with vLLM), with European prices. Order-of-magnitude figures; the methodology is what matters.

Starting data:

Parameter	Value	Source
European cloud rental (Scaleway, per H100)	2.73 €/h	Scaleway
→ 8×H100 node on-demand	~21.84 €/h	8 × 2.73
Amortised on-prem cost (estimate)	~10–13 €/h	capex 24–36 months + opex
Throughput	2,800 tok/s	reference benchmark
Power per H100 (TDP)	up to 700 W	NVIDIA
Reference PUE	1.4	typical of a datacenter

Cost per million tokens. At 2,800 tok/s, hourly throughput is 2,800 × 3,600 / 10⁶ = 10.08 million tokens/hour. Therefore:

$$\text{CPM (cloud)} = \frac{21{.}84}{10{.}08} \approx 2{.}17 \text{ €/1M tok}$$

With the amortised on-prem cost (~11 €/h) it drops to ~1.09 €/1M tok, the difference between renting from a provider and amortising your own iron.

Energy per token. The power of the 8 cards is 8 × 700 = 5,600 W (board only). Over 2,800 tok/s:

$$\text{energy per token} = \frac{5,600}{2,800} = 2 \text{ J/token (board)}$$

Over a million tokens that is 2 × 10⁶ J = 0.56 kWh (GPU only). Applying PUE 1.4: ~0.78 kWh per million tokens. At French wholesale prices (~0.058 €/kWh) that is ~0.045 € of electricity per million tokens; in Germany (~0.091 €/kWh), ~0.071 €, some 57 % more, because of the grid, not the model.

The link. If a serving optimisation raised throughput from 2,800 to 4,200 tok/s (+50 %), the CPM would fall to ~1.45 €/1M tok and energy per token to ~1.33 J, both at once, through the same denominator. That is the identity, with numbers.

Energy is inside the cost

The second link: energy is not an axis separate from cost, it is a large component of it. 2026 reference data:

Figure	Value	Source
Electricity as a share of TCO (5 years)	30–40 % (>50 % at high density 24/7)	Spheron
Day-ahead wholesale France (Mar-2026)	~58 €/MWh (0.058 €/kWh)	TradingEconomics
Day-ahead wholesale Spain	~54 €/MWh (0.054 €/kWh)	TradingEconomics
Day-ahead wholesale Germany	~91 €/MWh (0.091 €/kWh)	TradingEconomics
Electricity cost/token at 80 % util	¼ of that at 20 % util	Spheron

A methodological warning: the wholesale price (day-ahead) is not what a company pays; industrial retail adds grid charges and taxes (roughly ×1.5–2). But the relative ordering holds: France and Spain, cheap; Germany, expensive.

The third link, utilisation, closes the triangle: power is consumed even when the GPU is idle, so spreading the fixed cost (energy included) over more tokens (more throughput, more occupancy) lowers all three axes at once. The electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %.

Sensitivity analysis: what moves the cost per token

Four levers dominate the CPM, and it is worth knowing their elasticity before defending any figure:

Lever	Change	Effect on CPM	Mechanism
Throughput	+50 % tok/s	−33 % CPM	direct denominator
Utilisation	20 % → 80 %	up to −75 % on the electricity portion	spreads fixed cost
Location (energy)	Germany → France	lowers the electricity portion (~37 % less €/kWh)	wholesale FR < DE
Amortise vs rent	21.84 → 11 €/h	−50 % CPM	removes the provider’s margin
Quantisation (FP8/INT4)	+throughput, −VRAM/KV	lowers CPM and J/token	same identity

The reading: the CPM is not a number, it is a function of utilisation, throughput, location (energy price) and ownership model. Any on-prem vs cloud comparison that does not fix those assumptions is not comparable, and that is exactly the job of the synthesis article.

Simplified TCO model (on-premise)

The real on-premise cost is not the GPU alone: it is amortised capex + opex, over a hardware cycle of 24–36 months. Reference structure of an inference node:

Component	Type	Typical weight	Notes
GPUs	capex	the bulk	amortise over 24–36 months
Server, network, BMC	capex	medium	NICs, switching, out-of-band management
Electricity	opex	30–50 % of TCO	depending on country (FR/ES cheap, DE expensive) and PUE
Cooling	opex	inside the PUE	PUE 1.2–1.6 typical
Maintenance and support	opex	medium	spares, contracts, RMA
Staff and operations	opex	variable	the hidden cost that cloud rental includes

The difference between cloud rental (21.84 €/h in the example) and the real amortised cost (~10–13 €/h) is this breakdown: the provider charges a margin for absorbing capex, cooling and operations. The break-even point appears when sustained volume exceeds ~2M tokens/day, with typical amortisation in 6–12 months (Spheron). Below that threshold, renting usually wins; above it, amortising. Article A8 develops the full model and synthesis S3 connects it with sizing.

Reference: the price of a GPU-hour in European sovereign cloud

The other side of the on-prem comparison is the price of renting the GPU. For a sovereign European platform, what matters is not AWS/GCP/Azure (subject to the US CLOUD Act), but the European providers:

Provider	H100 (€/hour)	Location	Sovereignty
Scaleway	from 2.73	Paris, Warsaw	EU jurisdiction, exempt from US CLOUD Act
OVHcloud	+40–45 % vs Scaleway	France, Germany	EU jurisdiction, exempt from US CLOUD Act
Hyperscalers (reference)	3–7 USD	global	subject to US CLOUD Act

The data: Scaleway offers the H100 from 2.73 €/h in European datacenters (Paris, Warsaw), with models and data under EU jurisdiction, immune to extraterritorial regulation; OVHcloud serves it from France and Germany, ~40–45 % more expensive than Scaleway as of March 2026. Both are exempt from the US CLOUD Act, the foundational sovereignty requirement for data subject to GDPR (Scaleway, DeployBase).

The implication for the proposal: the comparison is not only cost, it is cost + sovereignty. An on-prem cluster of your own or a European cloud keep the data under EU jurisdiction; a US hyperscaler does not, however cheap the GPU-hour comes out. That is the leg that the cost axis on its own does not capture, and that synthesis S2 quantifies.

Energy and carbon reference: France, Germany, Spain

Here the choice of European country changes both the electricity cost and the carbon per token. It is the most important table for a sovereign proposal:

Country	Wholesale (€/MWh, Mar-2026)	Grid carbon (gCO₂/kWh)	Profile
France	~58	~20–60	nuclear: cheap and very clean
Spain	~54	~150–170	renewables + gas: cheap, medium carbon
Germany	~91	~363 (2024 average)	coal/gas + renewables: expensive and dirty

Sources: day-ahead wholesale prices (TradingEconomics), German carbon ~363 gCO₂/kWh (2024 average, Statista via Wikipedia), nuclear ~17–35 gCO₂e/kWh (EEA), Spain ~167 gCO₂/kWh (2020). The PUE multiplies all of that energy: at 1.4, for every kWh of GPU the datacenter spends 1.4 kWh with cooling included.

The calculation, on the example node (0.78 kWh per million tokens, with PUE): in France (~40 gCO₂/kWh) it is ~31 gCO₂ per million tokens; in Germany (~363 gCO₂/kWh), ~283 gCO₂ per million, a factor of ~9×. That is, the same load served in France emits of the order of 9× less carbon per token than in Germany, and costs ~37 % less in electricity, without touching a single line of the stack, purely because of the country’s grid. Siting the cluster in France or Spain is at once a cost lever, a carbon lever and a compliance lever. Hourly grid intensity is obtained from ElectricityMaps, the reference source for these calculations.

Scale context: global datacenters consumed ~460 TWh in 2025 (≈1.8 % of the world’s electricity), and the IEA projects passing 800 TWh in 2028, which is why energy efficiency per token stops being a detail and becomes a design constraint, and why carbon enters the European regulatory equation.

Sovereignty and regulation: the axis that does not appear in the CPM

For a European platform there is a fourth factor that the three numerical axes do not capture, and that can invalidate the lowest cost figure: jurisdiction. Three fronts that turn “the cheapest GPU-hour” into “the cheapest GPU-hour that complies”:

Front	What it requires	Architectural implication
GDPR + US CLOUD Act	personal data under a jurisdiction accessible to US authorities is exposed	European cloud (Scaleway/OVHcloud) or on-prem keep the data in the EU
EU AI Act	obligations for high-risk systems (traceability, control, documentation)	on-prem control makes auditability and compliance easier
CSRD / emissions reporting	reporting the carbon footprint (scope 2)	a low-carbon grid (France) improves the reported figure

The key point for the proposal: a US hyperscaler can offer a competitive GPU-hour, but it puts the data under the US CLOUD Act, which for data subject to GDPR is a compliance risk, not a saving. That is why this series’ comparison is not made in USD against US providers: it is made in euros, against options that keep data sovereignty, on-prem of your own or European cloud. Cost, energy and carbon are measurable axes; sovereignty is the constraint that decides which options even enter the table. This links with the mapping of the EU AI Act onto the on-premise architecture and with the technical controls ENS × ISO 42001 × EU AI Act.

The operational conclusion: France and Spain not only come out cheaper and cleaner than Germany; being EU jurisdiction, they meet the sovereignty constraint that a US hyperscaler does not meet at any price. Three advantages that reinforce each other, and that the proposal can defend with the same table.

Methodology: how to read (and not fool yourself with) these figures

Five warnings that apply to all the data in this series:

Board vs node. The GPU’s power (700 W per H100) is the floor; the complete node (CPU, NICs, power supplies, fans) consumes more. Always compare the same boundary.
Wholesale vs retail. The day-ahead price is not what the company pays; industrial retail adds grid charges and taxes. Use whichever matches your contract.
Measured vs estimated. A wattmeter figure and an nvidia-smi estimate are not the same thing; the difference is quantified in the energy articles.
Amortisation assumptions. The on-prem cost/hour depends on the hardware cycle (24 vs 36 months) and on the assumed utilisation; changing those assumptions moves the CPM more than any serving optimisation.
Comparability. Two CPM or J/token figures are only comparable if model, precision, context length, load and country match. Most public comparisons (and in USD) do not fix those assumptions, which is why the series reproduces the ones that matter with its own harness and European data.

The combined scorecard

The destination of the series is a single board where each model/configuration/location is scored on all the axes at once, to decide on a Pareto frontier rather than on a single number. An illustrative example (order-of-magnitude figures) on the same 8×H100 node:

Candidate	CPM (€/1M)	Goodput (tok/s)	Energy (J/tok)	Carbon (gCO₂/1M)	Sovereignty
A · 70B FP16, on-prem France	~1.09	~2,800	~2.0	low (grid ~40 gCO₂/kWh)	EU
B · 70B FP8, on-prem France	~0.75	~4,000	~1.4	lower	EU
C · Scaleway on-demand	~2.17	~2,800	n/a	low (FR)	EU
D · US hyperscaler	~3.0	~2,800	n/a	depends on region	not EU

The reading: B dominates A on the three intrinsic axes thanks to FP8. C (European cloud) is more expensive per token but shifts capex and operations, while keeping sovereignty. D (hyperscaler) may be competitive on cost but loses sovereignty, an axis that for GDPR data is not negotiable. No isolated number decides: the choice comes from seeing the whole row.

The trap the board avoids: optimising a single axis. Each cell will be filled in by an article in the series with measured data.

Where each axis is measured: from design to production

Each axis is measured at three different moments, with different tools and increasing precision. Confusing them is a common source of figures that do not add up:

Axis	Design (estimate)	Benchmark (controlled)	Production (continuous)
Cost	datasheet, GPU-hour price	CPM measured over synthetic load	OpenCost/Kubecost per team and token
Performance	theoretical throughput (roofline)	GuideLLM/AIPerf with SLO	OTel/Langfuse over real traffic
Energy	TDP × hours	Zeus/MLPerf Power on the bench	Kepler/DCGM per pod in the cluster

The rule: a design figure serves for sizing, but not for defending a proposal; for that you need a reproducible benchmark figure, and to operate, a continuous production one. When someone disputes a number in the proposal, the answer is not “a blog says so”, but “here is the bench, here is the methodology, reproduce it”.

Glossary of units and conversions

Quantity	Unit	Conversion / reference
Energy	joule (J), watt-hour (Wh)	1 Wh = 3,600 J; 1 kWh = 3.6 MJ = 3,600,000 J
Power	watt (W)	1 W = 1 J/s; H100 ≈ 700 W TDP
Energy per token	J/token	= average power (W) ÷ throughput (tok/s)
Throughput	tok/s	tokens generated per second by the system
Cost per token	€/1M tok (CPM)	= cost/hour ÷ (tok/s × 3,600 / 10⁶)
Latency	ms	TTFT (first token), TPOT/ITL (between tokens)
PUE	ratio (≥1)	total DC energy ÷ compute energy
Carbon	gCO₂eq	= energy (kWh) × PUE × grid intensity (gCO₂/kWh)

A useful conversion to keep in your head: at a constant 700 W, an H100 consumes ~0.7 kWh per hour, ~16.8 kWh a day and ~6,130 kWh a year (board only, no PUE). Multiplied by the PUE and your country’s energy price (FR ~0.058, ES ~0.054, DE ~0.091 €/kWh wholesale), that number is the basis of the electricity portion of the TCO.

How the rest of the series fits together

The 28 articles group into four tracks; this is the map:

Track	What it contributes	Closes with
FinOps (8)	measuring and allocating cost, TCO model	cost/token comparable on-prem vs European cloud
LLM benchmarking (8)	measuring performance reproducibly	Pareto frontier of engines
Energy (8)	measuring watts and carbon per token	energy in the TCO and in European regulation
Synthesis (4)	crossing the three axes	scorecard, on-prem vs cloud, sizing, harness

The three articles that follow this one in the batch introduce each axis in depth: their frameworks and the state of the art. From there, each article adds a layer of data to the final scorecard.

The thread running through all 28 instalments is the one in this article: cost, performance and energy are a single problem with three faces, tied together by throughput and by the energy that lives inside the cost, and in Europe, by the country where the cluster sits. Anyone who optimises one face while ignoring the others will end up with a pretty number and a platform that does not hold up. The work of the series is to produce the figures, measured, dated, European and reproducible, that allow a sovereign architecture to be defended before any committee with the only answer that closes a technical debate: the data and the bench to reproduce it.

Sources

Scaleway · H100 GPU instance (€/h price, EU sovereignty) — https://www.scaleway.com/en/h100/
DeployBase · OVHcloud GPU Pricing (European sovereignty) — https://deploybase.ai/articles/ovhcloud-gpu-pricing
TradingEconomics · electricity price France / Germany / Spain — https://tradingeconomics.com/france/electricity-price
EEA · emission intensity of electricity in Europe — https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1
Electricity sector in Germany (carbon ~363 gCO₂/kWh) — https://en.wikipedia.org/wiki/Electricity_sector_in_Germany
Spheron · AI Inference Power & Cost Economics (2026) — https://www.spheron.network/blog/ai-inference-cost-economics-2026/
NVIDIA · H100 (TDP/specs) — https://www.nvidia.com/en-us/data-center/h100/

GPU/LLM FinOps: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The references are European (providers and prices from FR/DE/ES), as this is a sovereign proposal; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Second article in the data series, and the first pillar deep dive. Here the FinOps tooling for GPU/LLM infrastructure is inventoried with the detail an architecture decision needs: what metrics each tool handles, how it allocates cost internally, what GPU support it has, under what licence and pricing model it operates, and where its limits are. It is the natural continuation of the opening article, which established that cost per token is the unit that allows on-prem and cloud to be compared. No recommendations: the final choice is settled in the synthesis article with the Pareto table; here there are only facts and methodology.

Why GPU FinOps is a different problem

Classic cloud FinOps was born to split CPU, memory and storage, cheap and elastic resources. The GPU breaks three assumptions at once, and that is why it needs its own treatment:

It is expensive and discrete. An H100 costs on the order of 2.7 to 7 €/hour in European cloud (Scaleway from 2.73 €/h; OVHcloud somewhat more), or its depreciated capex on-prem; it is not a resource that gets split in trivial fractions. A 10 % allocation error across a GPU fleet is real money.
It is easily underused. Unlike the CPU, a GPU draws its power even when idle, and average occupancy in ungoverned clusters is notoriously low. The cost of idle, the GPU powered on without working, is waste number one, and it is invisible if it is not measured.
It is hard to attribute. Who owns the cost of a GPU shared by time-slicing or MIG across several pods from several teams? Without an allocation layer, GPU spend is a black hole nobody claims and nobody optimises.

The goal of GPU FinOps is to turn that black hole into an attributed, measured and actionable invoice: knowing what each team, each model and, in the end, each token costs.

The FinOps metrics

Metric	Definition	Unit
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	€ / 1M tok
Cost per request	cost charged to a complete request	€ / req
€/GPU-hour	hourly cost of a GPU (depreciated or rented)	€/h
Utilisation	fraction of the GPU actually used	% (MFU, GPU-hour util)
Cost of idle	GPU powered on without useful work	€/h wasted
Showback	reporting the cost to each team (without charging)	€ / team
Chargeback	charging the cost to each team	€ / team
Cost efficiency	real cost ÷ cost if it were at 100 %	%

The metric that closes the loop with the business is cost per token (or per request): it is the only one that can be compared across providers, across models and against the price of an external API. All GPU FinOps tooling exists to reach, one way or another, that number.

The three FinOps phases (FinOps Foundation)

The FinOps Foundation framework organises the work in a three-phase cycle. Applied to GPU:

Phase	Goal	Typical GPU action
Inform	visibility and allocation	measure cost per namespace/team/model/token
Optimize	reduce spend	rightsizing, spot, quotas, switch off the idle, quantise
Operate	continuous governance	budgets, idle alerts, automated chargeback

The common mistake is jumping to Optimize without having done Inform: GPUs get bought or replicas get adjusted without knowing where the money really goes. Allocation (the Inform phase) is the prerequisite for everything else, and it is where the tooling comes in.

How cost is allocated in Kubernetes: OpenCost’s mechanics

OpenCost is the de facto standard, so it is worth understanding how it allocates cost, because that defines what any tool on top can and cannot do. It is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project) (OpenCost · GitHub).

The model works at node level: it starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit CPU/GPU/RAM prices, OpenCost uses the ratio of a set of base prices (the provider’s marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem). This is key on-premise: you define the node cost (depreciated capex + opex) and OpenCost splits it.

It obtains utilisation by scraping Prometheus: kube-state-metrics, node-exporter and cAdvisor give it the real consumption per pod, and with that it allocates cost per cluster, node, namespace, controller, service or pod (OpenCost · exporter). For the GPU, the signal comes from DCGM (via the NVIDIA GPU Operator) exported to Prometheus, the same base as GPU observability. A common idle detection pattern: an alert when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, routed to the team that owns the namespace.

Frameworks, tool by tool

OpenCost — the allocation standard (CNCF, Apache 2.0)

What it measures: allocated cost of in-cluster resources (CPU, GPU, memory, volumes) by any Kubernetes dimension. Method: node-level pricing model + Prometheus scraping (above). GPU: yes, via DCGM. Licence: Apache 2.0, free. It can run as a metrics exporter to Prometheus with no further dependencies. Limit: it is the allocation layer, it brings no optimisation, governance or product unit economics; for that you put something on top of it.

Kubecost — the commercial product on OpenCost (IBM)

What it measures: what OpenCost does (it is built on it) plus enterprise capabilities. Kubecost 3.0 (2025) added GPU monitoring via NVIDIA DCGM and integration with IBM Turbonomic for automatic rightsizing, and extended its scope from Kubernetes to cloud service cost. IBM acquired Kubecost and integrated Kubecost/OpenCost into its FinOps Suite alongside Cloudability and Turbonomic (CloudZero · Kubecost vs OpenCost). Differentiator: rightsizing, governance, support. Limit: it is a commercial product; the bulk of the value over free OpenCost is the optimisation and enterprise layer.

CloudZero — unit economics and cost per product

What it measures: it maps cloud cost to features, products, teams and customers, not just to resources. Method: it ingests multi-cloud billing and models it in business dimensions. Differentiator: the deepest allocation and the unit economics approach (cost per business unit). Limit: less focused on intra-Kubernetes mechanics than OpenCost/Kubecost; it is the “cost to business” layer.

Vantage — multi-cloud with many integrations

What it measures: multi-cloud cost with more than 20 native integrations (AWS, Azure, GCP, Kubernetes, Snowflake, Datadog, OpenAI and so on). Differentiator: breadth of sources, including the invoice from LLM providers (OpenAI), which brings it closer to end-to-end AI cost. Limit: the depth of intra-cluster allocation is lower than that of K8s-native tools.

Finout — virtual tagging, fast deployment

What it measures: multi-cloud cost with virtual tagging: it applies cost labels without modifying the real resources, which allows spend that was not properly tagged at source to be allocated. Differentiator: fast deployment and flexible reallocation without touching the infra. Limit: like the others in its category, it depends on the quality of the billing data it ingests.

Comparison table

Tool	Scope	GPU	Licence / model	Differentiator	Layer
OpenCost	Kubernetes	Yes (DCGM)	Apache 2.0 (CNCF), free	allocation standard	resource
Kubecost	K8s + cloud	Yes (DCGM, 3.0)	commercial (IBM)	rightsizing, enterprise	resource+optim
CloudZero	multi-cloud	indirect	commercial	unit economics per product	business
Vantage	multi-cloud (20+)	via K8s/provider	commercial	breadth (incl. OpenAI)	business
Finout	multi-cloud	via K8s/provider	commercial	virtual tagging	business

Pricing models of commercial tooling

They are worth knowing because the cost of the FinOps tool is also FinOps:

Model	How it charges	Range
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

(CloudZero · FinOps Tools). The implication: at large cloud spend, a 1–3 % fixed-fee can exceed the savings-based model; at small spend with a lot of waste, savings-based aligns incentives. OpenCost, being free, changes the equation for whoever has a team to operate it.

FOCUS: the cost data standard

The cross-cutting problem of multi-source FinOps is that each provider bills in its own format. FOCUS (FinOps Open Cost and Usage Specification) is the open technical specification that defines requirements so that providers produce uniform billing datasets (FOCUS · FinOps Foundation). The committee ratified FOCUS v1.3 on 4 December 2025.

What is relevant for this series: at FinOps X 2026 the focus has been put on extending FOCUS to AI workloads, with token economics pushing the specification, and the expansion requests include AI, datacenter and SaaS/PaaS workloads (SiliconANGLE). In other words, the standard that normalises cloud cost is being stretched to cover AI cost per token. For an architecture proposal, betting on tools that emit and consume FOCUS is betting on future interoperability.

From resource to token: cost per token with a gateway

Resource allocation (OpenCost) gets as far as “this vLLM pod cost X €/hour”. To get to “this request from this team cost Y” you have to intercept the inference traffic. That is where the gateway comes in.

Tools such as LiteLLM sit between the application and the LLM provider/engine, and intercept every request to record tokens, latency and cost in real time. Layers above (for instance OpenLM’s AI Gateway) generate usage logs compatible with FOCUS (v1.0 to 1.3) and map spend to team, product, customer or feature, enabling showback or chargeback (OpenLM · token attribution). This links up with FinOps and multi-tenancy of the GPU cluster, where the gateway is the piece that splits cost between tenants.

Worked example: chargeback of a multi-tenant cluster

To see the two halves joined, a split over an example node (4×H100, depreciated cost 12 €/hour) shared by three teams via namespaces and MIG:

Team	GPU-hours allocated (OpenCost)	Hardware cost (€/h)	Tokens/day (gateway)	Cost/1M tok
A · chat product	50 %	6.00	8M	~0.75 €
B · nightly batch	30 %	3.60	3M	~1.20 €
C · experimentation	20 %	2.40	0.5M	~4.80 €

How it comes out: OpenCost splits the node’s 12 €/h according to the GPU-hours each namespace consumes (half for A → 6 €/h). The gateway supplies the tokens per team. Each one’s cost per million tokens is its hardware cost divided by its output, and it reveals something neither half would see alone: team C pays 6× more per token than A, not because its model is worse, but because its GPU is underused (a lot of allocated GPU-hours for few tokens). That 4.80 €/1M is the chargeback signal that triggers an optimisation conversation: either C raises its utilisation, or it releases the GPU. Without crossing allocation and tokens, that waste stays hidden in a cluster average.

The hidden cost: utilisation

Optimisation lever number one is not changing GPU, it is stopping paying for an idle GPU. As seen in the opening article, the electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %, and utilisation spreads all the fixed cost (capex + energy) over more tokens. Measuring idle is, therefore, the highest-return action of the Optimize phase:

Signal	Source	Typical threshold
Idle GPU	`DCGM_FI_DEV_GPU_UTIL`	< 10 % for > 15 min
Unused GPU memory	`DCGM_FI_DEV_FB_USED`	reserved but not used
Pods with no traffic	gateway metrics	0 requests, GPU allocated

Allocation (OpenCost) locates whose the idle GPU is; scheduling and co-residency (sharing the GPU: time-slicing, MPS and MIG) recover it. FinOps closes the loop: measure → attribute → optimise → govern.

Optimize: the saving levers (with data)

Once cost is allocated (Inform), the Optimize phase has a bounded repertoire of levers. Ordered by typical return:

Lever	Typical saving	Mechanism	Cost/risk
Recover idle	the largest	switch off/share the idle GPU	requires measuring utilisation
Quantisation (FP8/INT4)	raises throughput, lowers VRAM	more tokens per GPU-hour	possible quality cost
Reserved commitment	20–40 % over on-demand	reserve capacity	less elasticity
Spot/preemptible	the deepest discount	interruptible capacity	you have to tolerate cuts
Rightsizing	variable	match GPU type/count to the SLO	requires benchmarks
Autoscaling (HPA/KEDA)	variable	scale replicas with demand	tuning metrics

On commitment, the 2026 cloud data is conclusive: reserved plans give 20–40 % saving against on-demand, and spot the deepest discount in exchange for interruptibility; the price of the H100 fell 64–75 % between Q4 2024 and the start of 2026 (Spheron · GPU Cloud Pricing). Rightsizing and autoscaling connect with cluster scheduling: fitting the workload onto the right GPU and scaling replicas with demand are, at the same time, performance and cost levers. Quantisation appears here because it raises throughput, and, by the identity from the opening article, lowers cost and energy per token at the same time.

GPU FinOps maturity

A simple model to place where an organisation is, and what it is missing:

Level	State	What is missing to move up
0 · blind	aggregate invoice, no attribution	instrument OpenCost + DCGM
1 · visibility	cost per namespace/team	per-token measurement (gateway)
2 · unit economics	cost per token and per product	idle alerts and budgets
3 · optimisation	idle recovered, commitment, rightsizing	automated chargeback
4 · governance	chargeback + budgets + FOCUS	continuous improvement

Most organisations with GPUs are at level 0 or 1: they see a big invoice but do not know whose each euro is. The value jump is in reaching level 2, cost per token and per product, which is exactly where the tooling in this article stops being optional. The series’ FinOps track walks that ladder up to the full TCO model and the comparable cost/token that underpins the architecture proposal.

The minimum stack to reach level 2

Bringing the previous pieces together into a concrete toolchain, all open source, that takes you from blindness (level 0) to cost per token and per product (level 2):

Piece	Function	Alternative
NVIDIA GPU Operator + DCGM	exports GPU metrics (usage, memory, power)	—
Prometheus	stores the usage and cost series	VictoriaMetrics
OpenCost	allocates cost per resource and dimension	Kubecost (commercial)
Gateway (LiteLLM)	counts tokens per team/model	OpenLM AI Gateway
Grafana	cost, utilisation and idle panels	—

It is, deliberately, infrastructure many clusters already have for observability (DCGM, Prometheus, Grafana); GPU FinOps reuses that base and adds OpenCost and the gateway to it. It is not a new product, it is a layer on top of what exists.

KPIs to watch

KPI	What it indicates	Typical target
Average GPU utilisation	waste	>70–80 % sustained
Cost per 1M tokens (per model)	economic efficiency	compare against cloud rental
% of GPU-hours idle	money thrown away	minimise (<10–15 %)
Cost per team/product	attribution	fair split, no surprises
Deviation against budget	governance	alert before it is exceeded

These five KPIs are what turn GPU FinOps from a pretty dashboard into a decision tool: if you watch them, you know at all times what each thing costs and where the waste is.

State of the art 2026

Consolidation in Kubernetes: OpenCost is the CNCF allocation standard; Kubecost (IBM) the reference commercial product, already with GPU via DCGM and rightsizing through Turbonomic.
From resource to token: the strong platforms of 2026 track cost at token and GPU level, combining resource allocation (OpenCost) with gateway measurement (LiteLLM); it is the only route to a comparable cost/token for on-prem against cloud.
FOCUS v1.3 (Dec 2025) as the interoperability layer, extending to AI at FinOps X 2026: the cloud cost standard absorbs token economics.
Mature GPU FinOps: instrumentation leans on DCGM (the same base as observability), and idle detection by a utilisation threshold is common practice.

Cost per token: the bridge to the business decision

All the previous tooling exists to produce a number the business understands: cost per token (or per request). It is the one that lets you answer the three questions that underpin an architecture proposal:

Build or buy? On-prem cost/token (depreciated capex + opex, at real utilisation) against the price of an external API or of cloud rental. Below the volume threshold (~2M tokens/day) buying usually wins; above it, building.
How do you price a product? If a feature consumes N tokens per use and each million costs C, the marginal cost per use is N·C/10⁶, the basis of any margin.
Where is the waste? Cost/token per team (the chargeback example) points at whoever underuses the GPU before the aggregate invoice hides it.

The trap: comparing cost/token across scenarios without fixing the assumptions (utilisation, precision, ownership model). An on-prem cost/token calculated at 80 % utilisation is not comparable with one at 20 %, nor an FP16 with an FP8. That is why allocation (Inform) is not an end in itself: it is the raw material of a cost model with explicit assumptions, which is what the synthesis article turns into the “build vs buy” argument with defensible numbers. GPU FinOps is not accounting; it is the quantitative basis of the architecture decision.

Limits and traps (data-driven)

Allocation ≠ per-token measurement. OpenCost splits the resource cost; without a gateway you do not reach the request. They are two halves that have to be joined explicitly.
Badly set base prices = badly set allocation. On-prem, OpenCost splits the cost you declare for the node; if the depreciated capex/opex is wrong, the whole split is wrong. Input data quality rules.
Invisible idle. Without DCGM exported to Prometheus and without utilisation alerts, waste number one does not appear on any dashboard.
Cost of the tool. A 1–3 % fixed-fee on large spend is real money; compare the tooling cost with the saving it delivers (it is FinOps on top of FinOps).
Proprietary formats. Tools that do not emit/consume FOCUS tie you to their data model; in 2026 the robust bet is FOCUS interoperability.

The next article in the series (A2) goes into OpenCost in depth; this one fixes the map. With allocation settled, the rest of the FinOps track builds the TCO model and the cost/token the proposal needs.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · documentación on-prem (modelo de precio por nodo) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exporter de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
CloudZero · FinOps Tools: Definitive Guide (2026) — https://www.cloudzero.com/blog/finops-tools/
FOCUS · especificación (FinOps Foundation) — https://focus.finops.org/focus-specification/
SiliconANGLE · FOCUS y la economía de tokens de IA (FinOps X 2026) — https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/
OpenLM · atribución de tokens en tiempo real (LiteLLM + FOCUS) — https://www.openlm.com/enable-ai-finops-with-real-time-token-attribution/
Finout · Best AI Cost Observability Tools (2026) — https://www.finout.io/blog/best-ai-cost-observability-tools-in-2026

Cost-per-Token on lo0 — Tech Blog

The second cost vector of AI agents: durable execution with Temporal

The thesis: agents have two cost vectors, not one

The three expenses of a faulty agent

Vector 1: failures charged twice

Vector 2: uncontrolled retries

Vector 3: inflated context on recovery

A worked example: the cost of one failure

How Temporal works: workflows, activities and determinism

Durable execution: what it is and the landscape

The cost of durable execution itself

Observability: from manual post-mortem to real-time traceability

The complete equation

State of the art and papers

The sovereign angle

FastMCP and the tool layer

Patterns: human-in-the-loop and long sequences

How to measure the second vector

Limits and traps (data-driven)

Closing

Sources

From GPU-hour to cost per token: the metric that compares on-prem and cloud

What this article covers

The identity: from GPU-hour to token

The two halves of cost per token

The gateway: LiteLLM in depth

The self-hosted trap: LiteLLM does not know what your own token costs you

Configuring custom pricing: an example

Cost per token changes with throughput: read it off the sweep

Cost per request: not only per token

Cost per model and per tier

Attribution: from cost to team and product

A combined report example

Governing spend: budgets, caps and alerts

On-prem vs European cloud comparison (the reason for all of it)

From cost per token to product price

Detailed comparison: on-prem, European cloud and API

Prompt caching and effective cost

State of the art 2026

Limits and traps (data-driven)

Closing

See also

Sources

Energy per token in Spain: methodology, the electricity market and the cases of speculation

What this article covers

The identity: energy per token, and why the price rules

Methodology: measuring energy per token without fooling yourself

The Spanish electricity market: how the price is formed

The volatility: the price that triples in a year

The structure of the bill: wholesale is not what you pay

The 2021–2022 crisis: the origin of the problem

The cases of speculation and manipulation

The other speculation: hoarding grid capacity (and the new 2026 law)

The design debate: why marginal pricing rewards speculation

The Iberian exception: a patch with an expiry date

What it means for the electricity cost of a cluster

Strategies for not being left exposed

Solar self-consumption: the numbers for a cluster

Annual electricity TCO: three scenarios for a cluster in Spain

PPAs: the central lever (and its small print in 2026)

What a PPA is, in short

Prices today: historic lows

Why so cheap: cannibalisation

Spain, European leader in PPAs

The catch for a datacenter: annual energy ≠ firm power

The context: the datacenter boom in Spain

The sovereign and carbon angle

Guarantees of Origin: carbon has its market too

Hourly intensity: the Red Eléctrica figure

Energy per token, with the Spanish price, step by step

Limits and traps (data-driven)

Closing

See also

Sources

OpenCost in depth: how GPU cost is allocated in Kubernetes

What this article covers

What OpenCost is

The cost model: node-level allocation

Where the data comes from: Prometheus

Architecture: the pieces