GPU/LLM FinOps: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The references are European (providers and prices from FR/DE/ES), as this is a sovereign proposal; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Second article in the data series, and the first pillar deep dive. Here the FinOps tooling for GPU/LLM infrastructure is inventoried with the detail an architecture decision needs: what metrics each tool handles, how it allocates cost internally, what GPU support it has, under what licence and pricing model it operates, and where its limits are. It is the natural continuation of the opening article, which established that cost per token is the unit that allows on-prem and cloud to be compared. No recommendations: the final choice is settled in the synthesis article with the Pareto table; here there are only facts and methodology.

Why GPU FinOps is a different problem

Classic cloud FinOps was born to split CPU, memory and storage, cheap and elastic resources. The GPU breaks three assumptions at once, and that is why it needs its own treatment:

It is expensive and discrete. An H100 costs on the order of 2.7 to 7 €/hour in European cloud (Scaleway from 2.73 €/h; OVHcloud somewhat more), or its depreciated capex on-prem; it is not a resource that gets split in trivial fractions. A 10 % allocation error across a GPU fleet is real money.
It is easily underused. Unlike the CPU, a GPU draws its power even when idle, and average occupancy in ungoverned clusters is notoriously low. The cost of idle, the GPU powered on without working, is waste number one, and it is invisible if it is not measured.
It is hard to attribute. Who owns the cost of a GPU shared by time-slicing or MIG across several pods from several teams? Without an allocation layer, GPU spend is a black hole nobody claims and nobody optimises.

The goal of GPU FinOps is to turn that black hole into an attributed, measured and actionable invoice: knowing what each team, each model and, in the end, each token costs.

The FinOps metrics

Metric	Definition	Unit
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	€ / 1M tok
Cost per request	cost charged to a complete request	€ / req
€/GPU-hour	hourly cost of a GPU (depreciated or rented)	€/h
Utilisation	fraction of the GPU actually used	% (MFU, GPU-hour util)
Cost of idle	GPU powered on without useful work	€/h wasted
Showback	reporting the cost to each team (without charging)	€ / team
Chargeback	charging the cost to each team	€ / team
Cost efficiency	real cost ÷ cost if it were at 100 %	%

The metric that closes the loop with the business is cost per token (or per request): it is the only one that can be compared across providers, across models and against the price of an external API. All GPU FinOps tooling exists to reach, one way or another, that number.

The three FinOps phases (FinOps Foundation)

The FinOps Foundation framework organises the work in a three-phase cycle. Applied to GPU:

Phase	Goal	Typical GPU action
Inform	visibility and allocation	measure cost per namespace/team/model/token
Optimize	reduce spend	rightsizing, spot, quotas, switch off the idle, quantise
Operate	continuous governance	budgets, idle alerts, automated chargeback

The common mistake is jumping to Optimize without having done Inform: GPUs get bought or replicas get adjusted without knowing where the money really goes. Allocation (the Inform phase) is the prerequisite for everything else, and it is where the tooling comes in.

How cost is allocated in Kubernetes: OpenCost’s mechanics

OpenCost is the de facto standard, so it is worth understanding how it allocates cost, because that defines what any tool on top can and cannot do. It is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project) (OpenCost · GitHub).

The model works at node level: it starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit CPU/GPU/RAM prices, OpenCost uses the ratio of a set of base prices (the provider’s marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem). This is key on-premise: you define the node cost (depreciated capex + opex) and OpenCost splits it.

It obtains utilisation by scraping Prometheus: kube-state-metrics, node-exporter and cAdvisor give it the real consumption per pod, and with that it allocates cost per cluster, node, namespace, controller, service or pod (OpenCost · exporter). For the GPU, the signal comes from DCGM (via the NVIDIA GPU Operator) exported to Prometheus, the same base as GPU observability. A common idle detection pattern: an alert when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, routed to the team that owns the namespace.

Frameworks, tool by tool

OpenCost — the allocation standard (CNCF, Apache 2.0)

What it measures: allocated cost of in-cluster resources (CPU, GPU, memory, volumes) by any Kubernetes dimension. Method: node-level pricing model + Prometheus scraping (above). GPU: yes, via DCGM. Licence: Apache 2.0, free. It can run as a metrics exporter to Prometheus with no further dependencies. Limit: it is the allocation layer, it brings no optimisation, governance or product unit economics; for that you put something on top of it.

Kubecost — the commercial product on OpenCost (IBM)

What it measures: what OpenCost does (it is built on it) plus enterprise capabilities. Kubecost 3.0 (2025) added GPU monitoring via NVIDIA DCGM and integration with IBM Turbonomic for automatic rightsizing, and extended its scope from Kubernetes to cloud service cost. IBM acquired Kubecost and integrated Kubecost/OpenCost into its FinOps Suite alongside Cloudability and Turbonomic (CloudZero · Kubecost vs OpenCost). Differentiator: rightsizing, governance, support. Limit: it is a commercial product; the bulk of the value over free OpenCost is the optimisation and enterprise layer.

CloudZero — unit economics and cost per product

What it measures: it maps cloud cost to features, products, teams and customers, not just to resources. Method: it ingests multi-cloud billing and models it in business dimensions. Differentiator: the deepest allocation and the unit economics approach (cost per business unit). Limit: less focused on intra-Kubernetes mechanics than OpenCost/Kubecost; it is the “cost to business” layer.

Vantage — multi-cloud with many integrations

What it measures: multi-cloud cost with more than 20 native integrations (AWS, Azure, GCP, Kubernetes, Snowflake, Datadog, OpenAI and so on). Differentiator: breadth of sources, including the invoice from LLM providers (OpenAI), which brings it closer to end-to-end AI cost. Limit: the depth of intra-cluster allocation is lower than that of K8s-native tools.

Finout — virtual tagging, fast deployment

What it measures: multi-cloud cost with virtual tagging: it applies cost labels without modifying the real resources, which allows spend that was not properly tagged at source to be allocated. Differentiator: fast deployment and flexible reallocation without touching the infra. Limit: like the others in its category, it depends on the quality of the billing data it ingests.

Comparison table

Tool	Scope	GPU	Licence / model	Differentiator	Layer
OpenCost	Kubernetes	Yes (DCGM)	Apache 2.0 (CNCF), free	allocation standard	resource
Kubecost	K8s + cloud	Yes (DCGM, 3.0)	commercial (IBM)	rightsizing, enterprise	resource+optim
CloudZero	multi-cloud	indirect	commercial	unit economics per product	business
Vantage	multi-cloud (20+)	via K8s/provider	commercial	breadth (incl. OpenAI)	business
Finout	multi-cloud	via K8s/provider	commercial	virtual tagging	business

Pricing models of commercial tooling

They are worth knowing because the cost of the FinOps tool is also FinOps:

Model	How it charges	Range
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

(CloudZero · FinOps Tools). The implication: at large cloud spend, a 1–3 % fixed-fee can exceed the savings-based model; at small spend with a lot of waste, savings-based aligns incentives. OpenCost, being free, changes the equation for whoever has a team to operate it.

FOCUS: the cost data standard

The cross-cutting problem of multi-source FinOps is that each provider bills in its own format. FOCUS (FinOps Open Cost and Usage Specification) is the open technical specification that defines requirements so that providers produce uniform billing datasets (FOCUS · FinOps Foundation). The committee ratified FOCUS v1.3 on 4 December 2025.

What is relevant for this series: at FinOps X 2026 the focus has been put on extending FOCUS to AI workloads, with token economics pushing the specification, and the expansion requests include AI, datacenter and SaaS/PaaS workloads (SiliconANGLE). In other words, the standard that normalises cloud cost is being stretched to cover AI cost per token. For an architecture proposal, betting on tools that emit and consume FOCUS is betting on future interoperability.

From resource to token: cost per token with a gateway

Resource allocation (OpenCost) gets as far as “this vLLM pod cost X €/hour”. To get to “this request from this team cost Y” you have to intercept the inference traffic. That is where the gateway comes in.

Tools such as LiteLLM sit between the application and the LLM provider/engine, and intercept every request to record tokens, latency and cost in real time. Layers above (for instance OpenLM’s AI Gateway) generate usage logs compatible with FOCUS (v1.0 to 1.3) and map spend to team, product, customer or feature, enabling showback or chargeback (OpenLM · token attribution). This links up with FinOps and multi-tenancy of the GPU cluster, where the gateway is the piece that splits cost between tenants.

Worked example: chargeback of a multi-tenant cluster

To see the two halves joined, a split over an example node (4×H100, depreciated cost 12 €/hour) shared by three teams via namespaces and MIG:

Team	GPU-hours allocated (OpenCost)	Hardware cost (€/h)	Tokens/day (gateway)	Cost/1M tok
A · chat product	50 %	6.00	8M	~0.75 €
B · nightly batch	30 %	3.60	3M	~1.20 €
C · experimentation	20 %	2.40	0.5M	~4.80 €

How it comes out: OpenCost splits the node’s 12 €/h according to the GPU-hours each namespace consumes (half for A → 6 €/h). The gateway supplies the tokens per team. Each one’s cost per million tokens is its hardware cost divided by its output, and it reveals something neither half would see alone: team C pays 6× more per token than A, not because its model is worse, but because its GPU is underused (a lot of allocated GPU-hours for few tokens). That 4.80 €/1M is the chargeback signal that triggers an optimisation conversation: either C raises its utilisation, or it releases the GPU. Without crossing allocation and tokens, that waste stays hidden in a cluster average.

The hidden cost: utilisation

Optimisation lever number one is not changing GPU, it is stopping paying for an idle GPU. As seen in the opening article, the electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %, and utilisation spreads all the fixed cost (capex + energy) over more tokens. Measuring idle is, therefore, the highest-return action of the Optimize phase:

Signal	Source	Typical threshold
Idle GPU	`DCGM_FI_DEV_GPU_UTIL`	< 10 % for > 15 min
Unused GPU memory	`DCGM_FI_DEV_FB_USED`	reserved but not used
Pods with no traffic	gateway metrics	0 requests, GPU allocated

Allocation (OpenCost) locates whose the idle GPU is; scheduling and co-residency (sharing the GPU: time-slicing, MPS and MIG) recover it. FinOps closes the loop: measure → attribute → optimise → govern.

Optimize: the saving levers (with data)

Once cost is allocated (Inform), the Optimize phase has a bounded repertoire of levers. Ordered by typical return:

Lever	Typical saving	Mechanism	Cost/risk
Recover idle	the largest	switch off/share the idle GPU	requires measuring utilisation
Quantisation (FP8/INT4)	raises throughput, lowers VRAM	more tokens per GPU-hour	possible quality cost
Reserved commitment	20–40 % over on-demand	reserve capacity	less elasticity
Spot/preemptible	the deepest discount	interruptible capacity	you have to tolerate cuts
Rightsizing	variable	match GPU type/count to the SLO	requires benchmarks
Autoscaling (HPA/KEDA)	variable	scale replicas with demand	tuning metrics

On commitment, the 2026 cloud data is conclusive: reserved plans give 20–40 % saving against on-demand, and spot the deepest discount in exchange for interruptibility; the price of the H100 fell 64–75 % between Q4 2024 and the start of 2026 (Spheron · GPU Cloud Pricing). Rightsizing and autoscaling connect with cluster scheduling: fitting the workload onto the right GPU and scaling replicas with demand are, at the same time, performance and cost levers. Quantisation appears here because it raises throughput, and, by the identity from the opening article, lowers cost and energy per token at the same time.

GPU FinOps maturity

A simple model to place where an organisation is, and what it is missing:

Level	State	What is missing to move up
0 · blind	aggregate invoice, no attribution	instrument OpenCost + DCGM
1 · visibility	cost per namespace/team	per-token measurement (gateway)
2 · unit economics	cost per token and per product	idle alerts and budgets
3 · optimisation	idle recovered, commitment, rightsizing	automated chargeback
4 · governance	chargeback + budgets + FOCUS	continuous improvement

Most organisations with GPUs are at level 0 or 1: they see a big invoice but do not know whose each euro is. The value jump is in reaching level 2, cost per token and per product, which is exactly where the tooling in this article stops being optional. The series’ FinOps track walks that ladder up to the full TCO model and the comparable cost/token that underpins the architecture proposal.

The minimum stack to reach level 2

Bringing the previous pieces together into a concrete toolchain, all open source, that takes you from blindness (level 0) to cost per token and per product (level 2):

Piece	Function	Alternative
NVIDIA GPU Operator + DCGM	exports GPU metrics (usage, memory, power)	—
Prometheus	stores the usage and cost series	VictoriaMetrics
OpenCost	allocates cost per resource and dimension	Kubecost (commercial)
Gateway (LiteLLM)	counts tokens per team/model	OpenLM AI Gateway
Grafana	cost, utilisation and idle panels	—

It is, deliberately, infrastructure many clusters already have for observability (DCGM, Prometheus, Grafana); GPU FinOps reuses that base and adds OpenCost and the gateway to it. It is not a new product, it is a layer on top of what exists.

KPIs to watch

KPI	What it indicates	Typical target
Average GPU utilisation	waste	>70–80 % sustained
Cost per 1M tokens (per model)	economic efficiency	compare against cloud rental
% of GPU-hours idle	money thrown away	minimise (<10–15 %)
Cost per team/product	attribution	fair split, no surprises
Deviation against budget	governance	alert before it is exceeded

These five KPIs are what turn GPU FinOps from a pretty dashboard into a decision tool: if you watch them, you know at all times what each thing costs and where the waste is.

State of the art 2026

Consolidation in Kubernetes: OpenCost is the CNCF allocation standard; Kubecost (IBM) the reference commercial product, already with GPU via DCGM and rightsizing through Turbonomic.
From resource to token: the strong platforms of 2026 track cost at token and GPU level, combining resource allocation (OpenCost) with gateway measurement (LiteLLM); it is the only route to a comparable cost/token for on-prem against cloud.
FOCUS v1.3 (Dec 2025) as the interoperability layer, extending to AI at FinOps X 2026: the cloud cost standard absorbs token economics.
Mature GPU FinOps: instrumentation leans on DCGM (the same base as observability), and idle detection by a utilisation threshold is common practice.

Cost per token: the bridge to the business decision

All the previous tooling exists to produce a number the business understands: cost per token (or per request). It is the one that lets you answer the three questions that underpin an architecture proposal:

Build or buy? On-prem cost/token (depreciated capex + opex, at real utilisation) against the price of an external API or of cloud rental. Below the volume threshold (~2M tokens/day) buying usually wins; above it, building.
How do you price a product? If a feature consumes N tokens per use and each million costs C, the marginal cost per use is N·C/10⁶, the basis of any margin.
Where is the waste? Cost/token per team (the chargeback example) points at whoever underuses the GPU before the aggregate invoice hides it.

The trap: comparing cost/token across scenarios without fixing the assumptions (utilisation, precision, ownership model). An on-prem cost/token calculated at 80 % utilisation is not comparable with one at 20 %, nor an FP16 with an FP8. That is why allocation (Inform) is not an end in itself: it is the raw material of a cost model with explicit assumptions, which is what the synthesis article turns into the “build vs buy” argument with defensible numbers. GPU FinOps is not accounting; it is the quantitative basis of the architecture decision.

Limits and traps (data-driven)

Allocation ≠ per-token measurement. OpenCost splits the resource cost; without a gateway you do not reach the request. They are two halves that have to be joined explicitly.
Badly set base prices = badly set allocation. On-prem, OpenCost splits the cost you declare for the node; if the depreciated capex/opex is wrong, the whole split is wrong. Input data quality rules.
Invisible idle. Without DCGM exported to Prometheus and without utilisation alerts, waste number one does not appear on any dashboard.
Cost of the tool. A 1–3 % fixed-fee on large spend is real money; compare the tooling cost with the saving it delivers (it is FinOps on top of FinOps).
Proprietary formats. Tools that do not emit/consume FOCUS tie you to their data model; in 2026 the robust bet is FOCUS interoperability.

The next article in the series (A2) goes into OpenCost in depth; this one fixes the map. With allocation settled, the rest of the FinOps track builds the TCO model and the cost/token the proposal needs.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · documentación on-prem (modelo de precio por nodo) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exporter de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
CloudZero · FinOps Tools: Definitive Guide (2026) — https://www.cloudzero.com/blog/finops-tools/
FOCUS · especificación (FinOps Foundation) — https://focus.finops.org/focus-specification/
SiliconANGLE · FOCUS y la economía de tokens de IA (FinOps X 2026) — https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/
OpenLM · atribución de tokens en tiempo real (LiteLLM + FOCUS) — https://www.openlm.com/enable-ai-finops-with-real-time-token-attribution/
Finout · Best AI Cost Observability Tools (2026) — https://www.finout.io/blog/best-ai-cost-observability-tools-in-2026

Focus on lo0 — Tech Blog