Chargeback on lo0 — Tech Blog

GPU chargeback and showback in multi-tenancy: how to split cluster cost between teams

Mon, 15 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

A cluster of 4×H100 SXM at ~1.40 €/GPU-hour (depreciated capex plus energy) shared between three teams with different occupancy produces a monthly chargeback report of 3 rows and an additional idle row that nobody claimed. Without tooling, that idle cost vanishes, diluted into the total. With OpenCost + LiteLLM + Kueue, attribution operates on three orthogonal planes: the iron (OpenCost, €/GPU-hour per namespace/label), token consumption (LiteLLM, €/token per key/team/model) and the scheduler quota (Kueue, GPUs reserved per ClusterQueue). Crossing the three produces the number that goes to finance: team B consumed X million tokens at Y €/1M tok, the GPU cost it Z €, and it still has 2 GPUs borrowed from the cohort that will cost it W € if it holds on to them next month.

Showback vs chargeback: the FinOps Foundation definition

The distinction is not one of technology but of accounting formality (FinOps Foundation — Invoicing & Chargeback, FinOps Foundation — Data Analysis and Showback):

Concept	Definition	Moves money	Requires
Showback	visibility of consumption and its cost per team; the report arrives, the budget does not change	No	metrics + attribution
Chargeback	the cost is transferred to the team’s or product’s P&L as real expenditure	Yes	showback + accounting policy + financial system

Two points from the framework worth pinning down:

Showback is a requirement of any FinOps practice; chargeback is optional and depends on the organisation’s accounting policy supporting transfers between cost centres.
Neither is “more mature” than the other. The narrative that chargeback is the “grown-up” version is false according to the framework itself. The natural sequence is: showback → trust in the data → chargeback if policy allows it.

When to use each

Situation	Recommended mode
First attribution cycle; teams do not trust the data yet	Showback
Teams with their own budget in a financial system	Chargeback
Shared cluster with no P&L separation per team	Showback with visible idle cost
Teams with a committed GPU availability SLA	Chargeback (real reservation)
Research/experimentation workloads with no formal budget	Showback + threshold alert

GPU cost attribution with OpenCost

Allocation API: attribution parameters

The OpenCost /allocation API is the piece that turns Prometheus metrics into a cost report per dimension (OpenCost — API):

Parameter	Values	Use in multi-tenancy
`window`	`today`, `7d`, `lastmonth`, RFC3339 range	window of the monthly report
`aggregate`	`namespace`, `label:LABEL`, `annotation:NAME`, `pod`, `controller`	attribution dimension
`includeIdle`	`true` / `false`	adds an `__idle__` row to the report
`shareIdle`	`true` / `false`	distributes idle among the non-idle allocations (proportional to non-idle cost)
`idleByNode`	`true` / `false`	computes idle per node instead of per cluster
`filter`	`namespace:"llm-prod"`, `label:equipo:"datos"`	restrict to one specific team

Query for a monthly report per team, with idle visible as a separate row:

curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d resolution=10m

With shareIdle=false (the default), OpenCost returns idle as a separate __idle__ entry, which makes visible how much was paid for unused capacity. With shareIdle=true, that idle is distributed among the tenants in proportion to their non-idle cost: each absorbs its share of the waste, which is the right behaviour for a chargeback that penalises whoever is responsible for the idle.

Labelling inference pods

For aggregate=label:equipo to work, the inference pods need the right label. Two methods:

Label in the pod spec (the most direct):

# vLLM Deployment for the datos team
metadata:
 labels:
 equipo: datos
 producto: rag-prod
 modelo: llama-70b
 entorno: prod

Annotation (when the label is already taken by another convention):

# aggregate as an annotation
aggregate=annotation:finops.io/equipo

OpenCost supports aggregate=label:KEY and aggregate=annotation:KEY with the same syntax; the choice between the two depends on the cluster’s labelling convention.

Shared costs and idle: the three modes

API mode	Behaviour	When
`includeIdle=false` (default)	Ignores idle; total cost looks lower than it is	Never in production
`includeIdle=true, shareIdle=false`	Idle in a separate `__idle__` row	Showback: visibility of waste
`includeIdle=true, shareIdle=true`	Idle distributed among tenants proportionally	Chargeback: the team pays its share of idle
`shareIdle=true, idleByNode=true`	Idle distributed per node (finer-grained if there are dedicated nodes)	Chargeback with heterogeneous nodes

For infrastructure with nodes dedicated per team, idleByNode=true is fairer: the idle of a node dedicated to team A is not transferred to team B.

LiteLLM as the attribution point for token consumption

Virtual keys and teams

LiteLLM materialises token chargeback with four mechanisms (LiteLLM — Virtual Keys, LiteLLM — Setting Team Budgets, LiteLLM — Spend Tracking):

Mechanism	What it does
Virtual key with `max_budget`	monthly budget per key; the key is blocked once it is exhausted
`budget_duration`	reset window: `30d`, `7d`, `24h`; the proxy runs a daily cron that resets according to the duration
`team_id`	groups keys; spend accumulates in `LiteLLM_TeamTable` per team
`tags`	tag each request; they allow budgets per tag (cost centre)
Spend logs	one row per request with `team_id`, `model`, `prompt_tokens`, `completion_tokens`, `response_cost`

Configuring teams with a monthly budget

# litellm-config.yaml — per-team budgets with the declared on-prem cost
model_list:
 - model_name: llama-3-70b-onprem
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-svc:8000/v1
 input_cost_per_token: 0.00000140 # 1.40 €/1M tok (€/GPU-hour ÷ throughput)
 output_cost_per_token: 0.00000140
 - model_name: llama-3-8b-onprem
 litellm_params:
 model: openai/llama-3-8b
 api_base: http://vllm-8b-svc:8000/v1
 input_cost_per_token: 0.00000035 # 0.35 €/1M tok (1 GPU, higher throughput)
 output_cost_per_token: 0.00000035

litellm_settings:
 success_callback: ["langfuse"] # or any external logger

# Teams (via the /team/new API or config)
# POST /team/new
# {
# "team_alias": "equipo-datos",
# "max_budget": 800, # 800 € per month
# "budget_duration": "30d",
# "tpm_limit": 500000, # max tokens/min
# "rpm_limit": 1000 # max requests/min
# }

The input_cost_per_token key comes from the identity in article A4: the €/GPU-hour from OpenCost divided by the benchmark throughput. With that value, the response_cost of each request reflects the real cost of the on-prem iron for that model.

Endpoints for querying spend per team

# Team spend accumulated over the period
GET /team/info?team_id=equipo-datos

# Team spend per model (aggregated spend_logs table)
GET /global/spend/logs?team_id=equipo-datos&start_date=2026-06-01&end_date=2026-06-30

# Total spend across all teams
GET /global/spend/teams

The response includes spend (accumulated cost in the configured currency), total_tokens, prompt_tokens, completion_tokens and a per-model breakdown, the direct fields for the monthly chargeback report.

Joining €/GPU-hour (iron) with tokens per team

The two attribution planes, OpenCost (iron) and LiteLLM (tokens), are crossed with one key: the model_name and the namespace. The join happens outside the tools (in a BI pipeline or a SQL query over the LiteLLM database and the data exported from OpenCost):

coste_por_token_real = OpenCost.GPU_cost_namespace / LiteLLM.total_tokens_team

When the on-prem cost is properly declared in input_cost_per_token, LiteLLM already does this calculation internally and the response_cost is correct. The explicit join serves to validate: if the sum of LiteLLM’s response_cost per team does not match the GPU cost OpenCost assigns to the namespace, there is drift in input_cost_per_token that has to be corrected (changed throughput, a new optimisation, more replicas).

Kueue as the GPU budget/quota mechanism

ClusterQueue, LocalQueue and cohorts

Kueue introduces the quota-based scheduling layer on top of Kubernetes (Kueue — Cluster Queue, Kueue — Cohort, Kueue — Fair Sharing):

Object	Scope	What it does
`ResourceFlavor`	cluster	maps resources to a group of nodes (e.g. H100 nodes)
`ClusterQueue`	cluster	defines `nominalQuota`, `borrowingLimit`, `lendingLimit` per resource/flavor
`LocalQueue`	namespace	entry point for the team’s workloads; points to a `ClusterQueue`
`Cohort`	cluster	groups ClusterQueues that can lend quota to each other

Quota concepts

nominalQuota: GPUs guaranteed to the ClusterQueue at all times.
borrowingLimit: maximum additional GPUs it can borrow from the cohort when others are not using them.
lendingLimit: GPUs from its nominalQuota that it allows to be lent to others (if not specified, it can lend all unused ones).
Fair Sharing: a mechanism that orders pending workloads by the historical resource usage of their LocalQueue, giving preference to whoever has consumed less. Compatible with hierarchical cohorts since Kueue v0.11.

Example YAML: 3 teams in the llm-platform cohort

# ResourceFlavor mapping to the H100 SXM nodes
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
---
# ClusterQueue for the datos team — 2 guaranteed GPUs, can borrow up to 2
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-datos
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 queueingStrategy: BestEffortFIFO
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 2
 borrowingLimit: 2 # can use up to 4 GPUs in total
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# ClusterQueue for the ia team — 1 guaranteed GPU, lends what it does not use
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-ia
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 lendingLimit: 1 # lends its GPU when it is not using it
---
# ClusterQueue for the plataforma team — 1 guaranteed GPU, low priority
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-plataforma
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-plataforma
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 borrowingLimit: 3 # can use all 4 if the rest are free
---
# LocalQueue in each namespace — entry point for the workloads
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-datos

How Kueue materialises the “GPU budget”

FinOps concept	Kueue mechanism
Guaranteed budget	`nominalQuota`: the team always has these GPUs available
Maximum spend limit	`nominalQuota + borrowingLimit`: absolute ceiling of admissible GPUs
Lending idle capacity	cohort + `lendingLimit`: others take what this one does not use
Fair sharing between teams	`Fair Sharing` + `WorkloadPriorityClass`: those who have consumed most wait longest
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`: the owner of the lent GPU reclaims it by evicting lower-priority workloads

The nominalQuota is the expression of the budget in GPUs: if the team has 2 nominal GPUs and the price is 1.40 €/GPU-hour, its maximum guaranteed spend is

\[ \text{maximum nominal spend} = 2 \times 1.40 \times 720 = 2{,}016 \text{ €/month} \]

The borrowingLimit sets the potential overspend if it borrows from the cohort (and that borrowing can be charged with the same formula, multiplying the borrowed usage hours by 1.40 €/GPU-hour).

Cost model: an example with 4×H100 SXM and 3 teams

Node price

Generic hardware: a server with 4×H100 SXM 80 GB. 2026 market prices: a single H100 SXM costs between 27,000 and 40,000 € depending on the source (GMI Cloud — H100 GPU Pricing 2026); a 4×H100 server sits in the 120,000–180,000 € range including chassis, power supplies and NVLink. We use 140,000 € as a generic assumption for the complete server (4 GPUs plus infrastructure).

Component	Calculation	€/hour node
Depreciated capex (140,000 €, 4 years, 90 % availability)	140,000 ÷ (4 × 8,760 × 0.9)	~4.43
Energy (4 × 700 W TDP × PUE 1.4 × 0.12 €/kWh)	4 × 0.7 × 1.4 × 0.12	~0.47
Operations / networking / maintenance	estimate	~0.70
Total 4×H100 node		~5.60 €/h
Per GPU	5.60 ÷ 4	~1.40 €/GPU-hour

Cost-per-token formula for the 70B model (sustained throughput ~2,000 tok/s at TP=4):

$$\text{€/1M tokens} = \frac{5.60 \text{ €/h}}{2{,}000 \text{ tok/s} \times 3{,}600 \text{ s/h}} \times 10^6 \approx 0.78 \text{ €/1M tokens}$$

Monthly scenario: 3 teams, heterogeneous utilisation

Reference month (720 hours). The 4 GPUs are nominally assigned: 2 to Datos, 1 to IA, 1 to Plataforma. Real occupancy varies:

Team	Nominal GPUs	Average occupancy	GPU-hours used	Idle GPU-hours	GPU cost (€)
Datos	2	75 %	1,080	360	~1,512
IA	1	60 %	432	288	~605
Plataforma	1	35 %	252	468	~353
Cluster idle	—	—	—	1,116	~1,562
Node total	4	—	1,764	1,116	~4,032

Total node cost per month: 5.60 €/h × 720 h = 4,032 €. Cost per GPU-hour used: 5.60 ÷ 4 = 1.40 €.

With shareIdle=false (showback), the 1,562 € of idle appears as a separate row in the report and nobody pays for it directly. With shareIdle=true (proportional chargeback), it is split among the three in proportion to their non-idle cost: Datos absorbs ~857 €, IA ~333 €, Plataforma ~192 € of additional idle.

Token consumption per team (LiteLLM spend logs)

Team	Model	Tokens/month	Cost/token	Token cost (€)
Datos	llama-70b (TP=4)	380 M	0.78 €/1M	~296
IA	llama-8b (1 GPU)	210 M	0.35 €/1M	~74
Plataforma	llama-8b + batch	80 M	0.35 €/1M	~28

Monthly chargeback report (showback + tokens)

Team	Iron GPU cost (€)	Assigned idle cost (€)	LiteLLM token cost (€)	Total chargeable (€)
Datos	1,512	857	296	2,665
IA	605	333	74	1,012
Plataforma	353	192	28	573
Idle (without shareIdle)	1,562	—	—	visible, not charged

Attribution dimensions × policy table

Attribution dimension	Tool	Showback	Chargeback
namespace	OpenCost `aggregate=namespace`	`/allocation?aggregate=namespace&includeIdle=true`	`shareIdle=true` + transfer to P&L
pod label	OpenCost `aggregate=label:equipo`	report per label	same with shareIdle
virtual key / team	LiteLLM `/global/spend/teams`	token dashboard	hard `max_budget` per key
ClusterQueue (reserved GPU)	Kueue `nominalQuota`	observe usage vs quota	committed budget in GPUs
request tag	LiteLLM `tag_budgets`	per cost centre	budget per tag
pod annotation	OpenCost `aggregate=annotation:KEY`	report per annotation	same with shareIdle

End-to-end data flow

When the Datos team has exhausted its 2 nominal GPUs and borrows those of Plataforma (which is not using them), Kueue records that loan. As soon as Plataforma launches a new workload:

Kueue detects that cq-plataforma is below its nominalQuota.
With preemption.reclaimWithinCohort: LowerPriority, Kueue evicts the Datos workload that was on the borrowed GPU, if its priority is lower.
The evicted workload returns to the cq-datos queue and is readmitted once quota frees up.

Fair Sharing orders the pending queue by the LocalQueue’s accumulated historical usage: if Datos has consumed more GPU-hours than IA in the recent period, IA’s next workloads have admission priority over new ones from Datos. This implements an equitable split without blocking anyone permanently.

Configuring OpenCost for the node price

For the numbers in the previous section to be reproducible, the node price has to be declared explicitly (the default value underestimates on-prem GPU, issue #3781):

# OpenCost Helm values.yaml — on-prem 4×H100 node, 5.60 €/h
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 4xH100 SXM node"
 CPU: "0.025" # €/CPU-hour (minor component)
 RAM: "0.003" # €/GB-hour
 GPU: "1.40" # €/GPU-hour ← the one that drives the split
 storage: "0.0002" # €/GB-hour

Verify after configuring:

# Check the price resolved per node
curl http://localhost:9003/allNodePricing

# Attribution query per team label, last month, idle separate
curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d accumulate=true

Reference PromQL for chargeback dashboards

# GPU cost assigned per team (label:equipo) — €/hour
sum by (label_equipo) (
 container_gpu_allocation
 * on(node) group_left node_gpu_hourly_cost
)

# Idle GPU-hours per node (idle > 15 min)
sum by (node) (
 (1 - avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) / 100)
 * on(node) group_left node_gpu_hourly_cost
)

# Accumulated monthly cost per team (sum over window)
sum_over_time(
 sum by (label_equipo) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
 )[30d:1h]
)

Sources

FinOps Foundation — Invoicing & Chargeback Capability — https://www.finops.org/framework/capabilities/invoicing-chargeback/
FinOps Foundation — Data Analysis and Showback — https://www.finops.org/framework/previous-capabilities/analysis-showback/
FinOps Foundation — Allocation Capability — https://www.finops.org/framework/capabilities/allocation/
OpenCost — API (Allocation API, parámetros window/aggregate/shareIdle/idleByNode) — https://opencost.io/docs/integrations/api/
OpenCost — API Examples — https://opencost.io/docs/integrations/api-examples/
OpenCost — GitHub (issue #3781, infra-precio GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
LiteLLM — Virtual Keys (max_budget, budget_duration, team_id) — https://docs.litellm.ai/docs/proxy/virtual_keys
LiteLLM — Setting Team Budgets — https://docs.litellm.ai/docs/proxy/team_budgets
LiteLLM — Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM — Budgets & Rate Limits — https://docs.litellm.ai/docs/proxy/users
LiteLLM — Setting Tag Budgets — https://docs.litellm.ai/docs/proxy/tag_budgets
LiteLLM — Budget Reset Times — https://docs.litellm.ai/docs/proxy/budget_reset_and_tz
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Administer Cluster Quotas — https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/
GMI Cloud — NVIDIA H100 GPU Pricing 2026 — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
IntuitionLabs — NVIDIA AI GPU Prices H100 Cost Guide — https://intuitionlabs.ai/articles/nvidia-ai-gpu-pricing-guide

From GPU-hour to cost per token: the metric that compares on-prem and cloud

Sun, 14 Jun 2026 03:30:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Fourth article in the FinOps track (A4), and the one that closes out the metric that gives the whole pillar its meaning: cost per token. In A2 we saw how OpenCost assigns the cost of the iron (the pod, in €/hour); in A3, which tool to use. But the cost of the pod is not the metric that compares on-prem with cloud, nor the one the business understands: that one is cost per token (or per request). This article explains how it is really calculated (spoiler: for your own inference it is not a list price, it is €/GPU-hour divided by throughput), how to instrument it with a gateway (LiteLLM), how to attribute it by team and product, and how to compare it against European cloud. No recommendations; only the mechanics and the numbers.

The identity: from GPU-hour to token

Cost per million tokens (CPM) is the cost of the iron divided by what it produces:

$$\text{cost per token} = \frac{\text{cost of the iron (€/h)}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

This is the central formula of the FinOps track, and it has a consequence many people never quite internalise: for a self-hosted model, the cost of a token does not exist as a fixed price. It depends on two things that change: the cost of the iron (from OpenCost, in €/hour) and the throughput (from your benchmark, in tok/s). The same model on the same node costs half as much per token if you double the throughput with an optimisation, because the denominator changes, not the numerator. The cost per token of your own inference is a function of your efficiency, not a tariff.

That sets it radically apart from an external API, where the price per token really is a fixed tariff the provider charges you. Comparing the two requires turning your variable cost into a number, and that is where the iron (A2) meets the tokens (this article).

The two halves of cost per token

Cost per token comes from joining two measurements that live in different places:

The iron half comes from OpenCost (article A2): the by-usage cost of the vLLM pod, in €/hour. The token half comes from the gateway, which intercepts every request and counts tokens per model and per team. Divide, and out comes the cost per token and, crucially, per team, which is what enables chargeback.

The gateway: LiteLLM in depth

The gateway sits between the application and the engine (vLLM) or the provider, and intercepts every request. LiteLLM is the OSS reference: it proxies over 100 models and tracks spend by keys, users and teams, logging spend for all known models automatically (LiteLLM · Spend Tracking).

Its key mechanisms:

Mechanism	What it does
`response_cost`	the cost of each response, available in logging (`kwargs["response_cost"]`)
Spend logs	one row per request with tokens, model, user, key and cost (SQLite/PostgreSQL)
Tags	tag each request by team, project, environment and model tier
Budgets	budgets and rate limits per team or user
Custom callbacks	bespoke loggers for key/user/model/tokens/cost

With the spend logs table (via the database adapter) you have the detail of every request to build cost dashboards (LiteLLM · Logging), and you can tag each request with team/project/environment for per-team analysis, with notifications (Slack) and hard caps for keys that must not exceed a limit. In practice it is the platform’s token meter and cost allocator.

The self-hosted trap: LiteLLM does not know what your own token costs you

Here is the point almost nobody wires up correctly. LiteLLM tracks cost using list prices for known models (the API ones: OpenAI, Anthropic and so on). But for a model self-hosted on your vLLM there is no list price, because you pay nobody per token. If you tell it nothing, LiteLLM does not know the real cost of your own token. The solution is custom pricing: declaring to LiteLLM the cost per token of your self-hosted model (LiteLLM · Custom Pricing).

And where does that number come from? From the identity above: the cost of the iron (OpenCost) divided by the throughput (your benchmark). That is, the correct flow is:

OpenCost gives you the cost of the vLLM pod: ~11 €/hour (A2).
Your benchmark gives you the throughput: ~2,800 tok/s (track B).
The identity gives the cost per token: 11 ÷ (2,800 × 3,600 / 10⁶) ≈ 1.09 €/1M tok.
That 1.09 €/1M goes into LiteLLM as custom pricing.

Only then does the response_cost of each request reflect the real cost of your inference, rather than a zero or somebody else’s list price. Without this step, your LiteLLM cost dashboards lie about your own models. It is the explicit junction of the three tracks: iron (FinOps), throughput (benchmarking) and measurement (gateway).

Configuring custom pricing: an example

In LiteLLM, the real cost of a self-hosted model is declared in its configuration. An example with the cost per token derived from the identity (1.09 €/1M ÷ 10⁶ = 0.00000109 €/token):

# LiteLLM config.yaml — real cost of a self-hosted model
model_list:
 - model_name: llama-3.1-70b-onprem
 litellm_params:
 model: openai/llama-3.1-70b  # your vLLM endpoint
 api_base: http://vllm:8000/v1
 input_cost_per_token: 0.00000109 # €/token (from OpenCost ÷ throughput)
 output_cost_per_token: 0.00000109 # refine if you separate prefill/decode

With that, every response_cost LiteLLM records reflects the real cost of your iron. The operational note: this number must be updated when the cost of the node changes (new depreciation, a different energy price) or the throughput does (an engine optimisation). It is not a constant; it is the output of OpenCost and of the benchmark, and it is recalculated whenever either of the two changes. Automating that recalculation, from node cost and measured throughput to custom pricing, is what keeps cost dashboards honest.

Cost per token changes with throughput: read it off the sweep

Because throughput is the denominator, cost per token is not a point, it is a curve over the concurrency sweep. On the ~11 €/hour node:

Concurrency	Throughput (tok/s)	Cost per 1M tokens
1	350	~8.73 €
8	2,100	~1.46 €
16	3,330 (goodput)	~0.92 €
24	3,900 raw / 2,420 goodput	~0.78 € raw / ~1.26 € on goodput

Important readings: at low concurrency the cost per token is sky-high (the fixed cost of the node is shared among few tokens); cost falls as occupancy rises. But watch the last row: if you compute cost on raw throughput (3,900) you get 0.78 €, whereas if you compute it on goodput (2,420, the part that meets the SLO) you get 1.26 €, and the latter is the real cost, because requests that violate the SLO do not count as useful production. An honest cost per token is computed on goodput, not on catalogue throughput. This is where the cost track meets the benchmarking one: the correct denominator is goodput.

Cost per request: not only per token

Cost per token is the unit; cost per request is what a complete request bills. A request consumes input tokens (prompt) plus output tokens (generation), and both count:

$$\text{cost per request} = (\text{input tokens} + \text{output tokens}) \times \text{cost per token}$$

With nuances: in many engines the prefill (input) and the decode (output) have different costs per token (decode is memory-bound and more expensive per token), so a fine-grained cost model distinguishes the two. For most cases a weighted average cost per token will do; to optimise, it pays to separate prefill and decode. LiteLLM records input and output tokens separately in its spend logs, so the data is there to refine with.

Cost per request is what connects to the product: if a feature consumes 1,500 tokens on average (1,200 of prompt plus 300 of output) and the cost is 1.09 €/1M, each use costs ~0.0016 €, the basis for setting margin, as seen in the FinOps introduction.

Cost per model and per tier

A platform does not serve a single model. Cost per token varies by model and by configuration, and the gateway tells them apart. An 8B and a 70B on the same node have very different costs per token: the 8B yields far more throughput (more tokens/s) for the same GPU-hour, so its cost per token is a fraction of the 70B’s. The design consequence: routing each request to the cheapest model that meets the quality bar is a direct cost lever, since serving a 70B where an 8B suffices means overpaying per token.

Model (example, same node)	Relative throughput	Relative cost per token
8B	high	low
70B FP8	medium	medium
70B FP16	low	high

The gateway, with its per-model custom pricing, makes this difference visible and enables cost-based routing: classify the request and send it to the appropriate tier. It connects with the L7 router of the serving track: the gateway not only measures cost, it can decide on the basis of it. And quantisation (FP16→FP8) shows up again as a lever: the same model in FP8 raises throughput and lowers cost per token, moving the figure without changing model.

Attribution: from cost to team and product

The reason for putting a gateway in place is not only to measure total cost, it is to share it out. By tagging each request with team, product and environment, LiteLLM builds the split by business dimension:

With per-team budgets and hard caps per key, the gateway does not only report: it governs. A team that goes over budget gets an alert; a key with a hard cap cannot spend any more. It is the FinOps Operate phase applied to the token.

A combined report example

Joining the iron (OpenCost) and the tokens (gateway), the monthly cost-per-token-per-team report on the example node:

Team	Tokens/month	Applied cost per token	Charged cost
A · chat-prod	240M	1.09 €/1M (70B FP8)	~262 €
B · batch	90M	0.40 €/1M (8B)	~36 €
C · experimentation	15M	1.09 €/1M (70B)	~16 € + idle

What it reveals: A consumes a lot but at an efficient cost (FP8, high utilisation); B uses a cheap model (8B) for its workload; C consumes little but, as we saw in A2, drags along the idle cost of its underused GPU. The token report plus the OpenCost iron report give the full picture: who spends, on which model, with what efficiency. It is the chargeback that neither half would produce alone. And the per-team figure is what gets taken into the budget conversation: not “the cluster costs X”, but “your team consumed Y million tokens at Z €/million”.

Governing spend: budgets, caps and alerts

Measuring without governing is a dashboard; the value lies in closing the loop. LiteLLM allows three levels of control over spend:

Control	What it does	When
Soft budget	alert (Slack) when a threshold is crossed	tracking teams
Rate limit	limits requests/tokens per minute	avoiding cost spikes
Hard cap	the key cannot spend any more	keys that must never overrun

The healthy pattern: soft budgets with alerts for the teams (so they adjust their consumption), rate limits to smooth spikes, and hard caps only on critical keys (a service that must not run away). With this, cost per token stops being an after-the-fact figure and becomes a real-time control: the FinOps Operate phase, applied to inference.

On-prem vs European cloud comparison (the reason for all of it)

Cost per token exists to answer one question: is it cheaper to serve on my own iron or to rent? With the identity and European figures:

Option	Cost per 1M tokens	Notes
Depreciated on-prem (8×H100, 2,800 tok/s)	~1.09 €	your iron, your sovereignty
European cloud Scaleway (8×H100, on-demand)	~2.17 €	EU, no iron to operate
External API (proprietary model)	provider tariff	no data sovereignty

Cost per token is what makes the incomparable comparable: your variable cost (iron ÷ throughput) against a provider’s fixed tariff. And it exposes the lever: if an optimisation raises your throughput from 2,800 to 4,200 tok/s, your on-prem cost drops to ~0.73 €/1M, widening the advantage over renting. That is why benchmarking (track B) is, indirectly, the tool that moves cost per token the most: every goodput improvement translates into this number.

From cost per token to product price

Cost per token closes the loop with the business when it becomes cost per unit of product. The chain: cost per token (from the identity) × tokens per use of a feature = marginal cost of that feature. With that, three business decisions become data-backed:

Setting a price. If serving a feature costs 0.0016 € per use, the price to the customer has to cover it with margin; cost per token is the floor of pricing.
Spotting loss-making features. A feature that consumes many tokens (long prompts, lots of output) may cost more than it contributes; cost per request reveals it before you scale.
Comparing build vs buy at product level. On-prem cost per token against an API tariff, for that specific workload, decides whether to serve in house or consume externally.

This is the ultimate reason for the whole FinOps track: it is not infrastructure accounting, it is the quantitative basis for product and architecture decisions. A measured and attributed cost per token turns “is this profitable?” from an intuition into a calculation.

Detailed comparison: on-prem, European cloud and API

Extending the comparison with scenarios, for the same job (1M tokens):

Option	Cost/1M tok	Sovereignty	Where it fits
Depreciated on-prem, high utilisation	~0.7–1.1 €	full (EU)	high, sustained volume
Depreciated on-prem, low utilisation	~2–4 €	full (EU)	bad: idle drives cost up
European cloud (Scaleway, on-demand)	~2.2 €	EU	variable volume, no iron to operate
European cloud (reserved/committed)	~1.5–1.8 €	EU	medium committed volume
External proprietary API	provider tariff	non-EU	prototype / no sovereignty requirement

The table shows two things. First, on-prem only wins if utilisation is high, since at low occupancy idle makes it dearer than cloud, which connects cost per token with scheduling and utilisation (the idle of A2). Second, the external API, however competitive its tariff, loses data sovereignty, an axis that for GDPR data is not negotiable. Cost per token is the metric that puts the first three on the same scale; sovereignty is the constraint that rules out the last one for certain kinds of data.

Prompt caching and effective cost

One nuance refines the real cost per token: prefix caching. When many requests share the same prefix (a long system prompt, a repeated context), the engine caches its prefill compute and does not recompute it on every request. That means cached input tokens cost almost zero the second time and thereafter, so the effective cost per token of a workload with a high cache hit rate is lower than the nominal cost. External APIs already reflect this with a reduced price for cached tokens; in self-hosted the effect is that throughput rises (less prefill to recompute), which by the identity lowers cost per token.

The implication for the cost model: if your workload has shared prefixes (RAG over the same corpus, agents with the same system prompt), the real cost per token is lower than the naive calculation gives, and it pays to measure the prefix caching hit rate so as not to overestimate cost. And the reverse: a workload of unique, never-repeated prompts gains nothing from the cache, and its cost per token is the full nominal figure. Cost per token, once again, is not a constant: it depends on the model, the precision, the utilisation, the goodput and the reuse pattern of the workload. Measuring it on your real traffic, not on a synthetic benchmark without a cache, is what gives the number you actually pay.

State of the art 2026

The gateway as standard: LiteLLM (and alternatives) are consolidating as the layer that measures tokens, attributes cost and governs budgets in the LLM stack.
FOCUS for tokens: layers on top of the gateway generate FOCUS-compatible logs, integrating AI cost with the rest of cloud spend (FinOps introduction).
Custom pricing as a bridge: the OpenCost→gateway connection via custom pricing is the standard way to give self-hosted models a real cost.
Cost per token as a business KPI: increasingly, cost per token (and per product) is the metric the business demands, not the cost of the pod.

Limits and traps (data-driven)

Forgotten custom pricing. Without declaring to LiteLLM the real cost of your self-hosted token, the response_cost of your own models is zero or someone else’s price. Close the OpenCost→gateway bridge.
Static throughput. Cost per token changes with throughput; if you optimise the engine but do not update the custom pricing, the reported cost drifts out of date.
Forgetting prefill/decode. An average cost per token is fine for reporting; to optimise, separate input and output.
Measuring without governing. Spend logs without budgets or caps make a pretty dashboard; the value lies in closing the loop with alerts and caps.
Comparing tariffs with costs without matching assumptions. Your 1.09 €/1M assumes a utilisation and a throughput; the provider’s tariff does not. Fix the assumptions before comparing.

With cost per token solved, the FinOps track has its central metric; the following articles (A5 multi-tenant chargeback, A8 TCO model) build on it. The full chain, iron (OpenCost) ÷ throughput (benchmark) = cost per token, measured and attributed by the gateway, is what turns the cost of AI into a defensible number, in euros and per team.

Closing

Cost per token is the metric that reconciles the three tracks of the series into a single number, and its underlying lesson is counter-intuitive: for your own inference, the cost of a token is not a price, it is the reflection of your efficiency. It comes from dividing the cost of the iron (OpenCost) by useful throughput (the goodput of your benchmark), and it becomes visible and governable with a gateway (LiteLLM) that counts the tokens, attributes them by team and applies budgets. The mistake that invalidates the whole exercise is leaving custom pricing at zero and believing your own inference is “free”; the right move is to close the OpenCost→gateway bridge and keep the number up to date when the node cost or the throughput changes. Done that way, cost per token answers with data the one question the business keeps repeating, is it worth serving this on my own iron, and it does so in euros, per team and per product, comparable against European cloud and against any API. For a sovereign architecture proposal, it is the column of the dashboard that translates all the cost engineering into the sentence that closes a budget meeting: this is the cost per token, this is how it is measured, and here is the bench to reproduce it.

Sources

LiteLLM · Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM · Custom Pricing (modelos self-hosted) — https://docs.litellm.ai/docs/proxy/custom_pricing
LiteLLM · Logging — https://docs.litellm.ai/docs/proxy/logging
LiteLLM · documentación general — https://docs.litellm.ai/docs/
Statsig · LiteLLM cost tracking — https://www.statsig.com/perspectives/litellm-cost-tracking
StackPulsar · LiteLLM production monitoring (2026) — https://stackpulsar.com/blog/litellm-production-monitoring/

FinOps and multi-tenancy on the GPU cluster: who pays for what

Thu, 11 Jun 2026 12:20:00 +0000

Third instalment in an operational series on how to squeeze a generic on-premise LLM inference cluster of 4×H100 SXM 80 GB. The sibling pieces are Multimodal VLM on-premise with vLLM, serving vision-language models on the same cluster, and Speeding up cold start with Tensorizer, cutting the model load time from disk to HBM. This post closes the economic question the other two leave open: once the cluster serves several models for several teams, who pays for what?

TL;DR

An on-premise GPU cluster does not have the property that makes FinOps easy in the cloud: no monthly invoice arrives telling you what each compute hour cost. The cost has to be manufactured from capex and energy. The right unit is the token, and it is derived in two steps. First the €/GPU-hour: capex depreciation spread over the hours of useful life, plus energy (power × PUE × price_kWh). For an H100 SXM with an assumed capex of 30,000 € and 4 years of life at 90 % availability, that gives ≈ 1.07 €/GPU-hour (0.95 of depreciation + 0.12 of energy at 0.12 €/kWh and PUE 1.4). Second, the €/1M tokens: divide that €/GPU-hour by the sustained throughput. At 2,000 useful tok/s, ≈ 0.148 €/1M tokens; but that number only holds if the GPU is at 100 % useful utilisation. At 20 % utilisation the same token costs 4× more (0.74 €/1M), because capex and energy keep being paid even if the GPU is idle. That is the central thesis: the FinOps goal is not to negotiate the price of the token down, there is no provider to negotiate with, but to raise useful utilisation, which is the only term that moves cost by an order of magnitude. Attribution (chargeback / showback) is made real with LiteLLM virtual keys: per-team and per-user budget, RPM/TPM rate limit, automatic spend tracking and tags to assign spend to cost centres. Isolation between tenants is achieved with MIG (hard GPU partitioning), namespaces, ResourceQuota and PriorityClass. The FinOps Foundation rule worth carving in stone: showback always, chargeback only if the organisation’s accounting policy supports it; neither of the two is “more mature”. And the governance metric of the shared cluster is not the €/token but DCGM_FI_DEV_GPU_UTIL.

The problem the cloud hides from you and on-premise forces you to solve

In the cloud, the cost of a GPU is an observable fact: AWS, GCP or Azure charge you per GPU-hour and the invoice arrives itemised. FinOps in that world consists of reading an invoice that already exists and attributing it. On-premise there is no such invoice. The GPU was bought once, eighteen months ago, on a purchase order nobody remembers; the electricity bill arrives for the whole building with no breakdown of what fraction was the cluster’s; and the datacenter PUE is a number the facilities team knows but the AI team never asked about.

The first job of on-premise FinOps, then, is not to attribute a cost but to manufacture one. You have to build, from explicit assumptions, the equivalent of the “price per GPU-hour” that in the cloud is given to you. And once manufactured, you have to understand that this number is not a property of the hardware, like VRAM or TDP, but a function of how the hardware is used. An idle H100 costs exactly the same as an H100 at 100 %: the capex is already paid and idle energy is not zero. The only thing that changes is how many useful tokens that fixed cost produced. That is where the whole thesis of this post comes from.

The analogy: the coworking space with metered services

A coworking space manages a finite physical space and rents it to teams. The cost model has exactly the structure of the GPU cluster, and the analogy holds down to the last detail.

The desk-hour is the spread capex. The manager paid the rent on the premises, the furniture and the refurbishment, a one-off outlay or a fixed monthly cost, and spreads it over the available desk-hours in the month. Each desk has a base cost that does not depend on whether anybody uses it: the walls, the table and the chair cost the same empty as occupied. This is the depreciation of the GPU: capex divided by the hours of useful life.

The metered kWh is the energy. On top of the desk-hour, the coworking space measures the actual electricity consumption, the plugged-in laptop, the screen, the air conditioning in that room, and bills it separately. This is the energy of the GPU: power × hours × price_kWh, multiplied by the datacenter’s PUE, which is the coworking space also charging you for the fraction of the building’s air conditioning that cools your room.

The empty desk is paid for by whoever booked it. Here is the heart of the matter. If a team books ten desks for the whole month and only uses two, it pays for all ten anyway. The empty desk still takes up space the manager cannot resell, still depreciates furniture, still counts as committed capacity. In the cluster: a GPU assigned to a tenant that uses it at 20 % costs the same as if it used it at 100 %, and the idle 80 % is wasted capex that somebody is paying for.

The manager optimises occupancy, not tariff. A competent coworking manager does not obsess over lowering the price of the desk-hour, it is a sunk cost, the premises are already paid for. They obsess over the occupancy rate: every empty desk is margin lost forever, because an unsold desk-hour cannot be stored. The FinOps equivalent: the goal is not to lower the €/token, it is to raise utilisation, because the €/token falls only as a consequence of utilisation rising. There is no provider to squeeze; the only margin lies in not leaving GPUs idle.

The analogy has an honest limit worth naming: in a coworking space, two people cannot physically share the same chair. On a GPU you can, with time-slicing, MPS or MIG (see Sharing a GPU), and that is precisely the technical lever that lets you raise occupancy beyond what the physical coworking space would allow. But the economics of fixed cost + metered cost is identical.

The token as the unit of cost

Why the token and not the GPU-hour, or the request, or the model? Because the token is the only unit comparable across heterogeneous workloads. A GPU-hour of Llama 8B and one of Llama 70B produce radically different amounts of “useful work”; a RAG request with a 50-token answer and one generating a 3,000-token report are not comparable. The token, specifically the pair (input_tokens, output_tokens), normalises all of that. It is the unit LiteLLM accounts for natively, the one that appears in the OTel attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (see LLM tracing with OpenTelemetry GenAI), and the one commercial providers already use to charge. Adopting the token as the internal currency makes on-premise chargeback directly comparable with the cloud alternative, which is exactly the comparison management wants to see.

The two derived metrics that matter:

Tokens/€ — how many useful tokens each euro of total cluster cost buys. It rises when utilisation rises. It is the metric of economic efficiency.
Tokens/W — how many tokens each watt consumed produces. It is the metric of energy efficiency and, in a datacenter with limited power (the usual case on-premise), it is often the real hard constraint: you cannot add more GPUs because there are no more kW in the rack, so every watt has to perform.

Cost model of a GPU: from capex to €/token

The hourly cost of a GPU has two addends. Let us put all the assumptions in writing, because, as in capacity planning, a cost calculation without written assumptions is a disposable calculation.

Depreciation (spread capex)

$$\text{depreciation cost/h} = \frac{\text{capex}_{\text{GPU}}}{\text{hours of useful life}}$$

where the hours of useful life are the depreciation years × 8,760 h/year × the effective availability (the GPU is not available 100 % of the time: there is maintenance, restarts, update windows). Explicit, generic assumptions (they are not figures from any provider or any real purchase):

Capex per GPU: 30,000 € — a reasonable assumption that includes not just the GPU but its pro-rata share of the server, power supply, cooling, NVLink/InfiniBand network and rack. A “bare” H100 SXM costs less, but honest FinOps spreads the cost of the whole node across its GPUs.
Accounting useful life: 4 years. That is aggressive-realistic for a datacenter GPU; some depreciate over 3, others over 5.
Effective availability: 90 %.

$$\text{hours of life} = 4 \times 8{,}760 \times 0.90 = 31{,}536 \text{ h}$$ $$\text{depreciation cost/h} = \frac{30{,}000}{31{,}536} \approx 0.95 \text{ €/GPU-hour}$$

Energy (power × PUE × price_kWh)

$$\text{energy cost/h} = P_{\text{GPU}} \cdot \text{PUE} \cdot \text{price kWh}$$

Assumptions:

GPU power: the TDP of the H100 SXM is 700 W according to the NVIDIA datasheet (H100 datasheet). Under sustained inference load it hovers around that value; we use 0.7 kW as the average power under load. (At idle it drops a lot, 60–100 W, but the FinOps of reserved capacity reasons about power under load, which is what limits the rack.)
PUE (Power Usage Effectiveness): how many watts enter the datacenter for every watt that reaches the chip. A modest, well-managed on-premise datacenter sits at 1.4; a bad one at 2.0; the hyperscalers boast of 1.1. We use 1.4.
Price per kWh: 0.12 €/kWh — a generic assumption for an industrial tariff; it varies enormously by country and contract.

$$\text{energy cost/h} = 0.7 \cdot 1.4 \cdot 0.12 \approx 0.1176 \approx 0.12 \text{ €/GPU-hour}$$

The total €/GPU-hour

$$\boxed{\text{€/GPU-hour} = 0.95 + 0.12 \approx 1.07 \text{ €/GPU-hour}}$$

Two important readings of this number. First: depreciation dominates (89 % of the cost); energy is 11 %. This inverts many people’s intuition, who believe “the expensive part is the electricity”. With these assumptions, the expensive part is having bought the GPU, and that is why leaving it idle hurts so much. Second: the number is per-GPU; for the 4×H100 node it is ≈ 4.28 €/hour, and for the complete generic cluster of 4 nodes (16 GPUs), ≈ 17.1 €/hour ≈ 150,000 €/year of fixed cost that is paid whether it is used or not.

From €/GPU-hour to €/token

Now we divide the hourly cost by the tokens the GPU produces in an hour. Here sustained throughput comes in. Let us suppose, an order-of-magnitude figure, checkable with vllm bench serve, that a Llama 70B replica on 4×H100 (TP=4) sustains 2,000 useful tokens/s aggregated under good concurrency.

$$\text{tokens/hour} = 2{,}000 \times 3{,}600 = 7.2 \times 10^6 \text{ tok/h}$$

The cost of that replica (4 GPUs) is $4 \times 1.07 = 4.28$ €/hour. Therefore:

$$\text{€/1M tokens} = \frac{4.28}{7.2} \approx 0.59 \text{ €/1M tokens}$$

Per individual GPU, if we take a smaller replica (e.g. Llama 8B FP8 on 1 GPU at ~2,000 tok/s):

$$\text{€/1M tokens} = \frac{1.07 \text{ €/h}}{7.2 \times 10^6 \text{ tok/h}} \times 10^6 \approx 0.149 \text{ €/1M tokens}$$

These numbers are the theoretical floor at 100 % utilisation. Nobody operates at 100 %. And that is the whole story.

Utilisation is the lever (with numbers)

The €/token in the previous section assumes the GPU produces 7.2 M tokens every hour, hour after hour. In practice it produces that only during peak hours. The rest of the time it is partly idle: at night, at weekends, between traffic peaks. The useful utilisation $u$ is the fraction of the theoretical capacity that really turns into billable tokens.

The hourly cost is fixed (1.07 €/GPU-hour is paid come rain or shine), but the tokens produced scale with $u$:

$$\text{€/1M tokens}(u) = \frac{\text{€/GPU-hour}}{\text{max tok/hour} \cdot u} = \frac{1.07}{7.2 \times 10^6 \cdot u} \times 10^6 = \frac{0.149}{u}$$

The requested contrast, worked out:

Useful utilisation $u$	Real tokens/h	€/1M tokens	Multiplier vs 80 %
100 %	7.2 M	0.149 €	0.80×
80 %	5.76 M	0.186 €	1.00× (baseline)
50 %	3.6 M	0.298 €	1.60×
20 %	1.44 M	0.744 €	4.00×
10 %	0.72 M	1.488 €	8.00×

Direct reading: going from 20 % to 80 % utilisation divides the cost per token by four ($0.744 \to 0.186$). No tariff negotiation, no quantization, no hardware change gives you that factor of 4 so cheaply. Quantizing from BF16 to FP8 can double throughput, and therefore halves the €/token, but it degrades quality and demands evals (see Quantization for inference); raising utilisation from 20 to 80 % does not touch the quality of a single token.

This reformulates GPU cluster FinOps in a single sentence: the €/token is not something you negotiate, it is something you earn by filling the GPUs. And raising utilisation is an engineering problem that already has its pieces:

Autoscaling that switches replicas off when traffic drops, so as not to pay for idle GPU (see LLM autoscaling on Kubernetes with KEDA). Switching off a 4-GPU replica for eight hours overnight saves ≈ 34 € a day of real cost; multiplied by replicas and by days, that is the difference between a profitable cluster and one that bleeds.
Sharing a GPU between small workloads with MIG or time-slicing, so that two tenants that individually would use 30 % together fill a GPU to 60 % (see Sharing a GPU).
Overnight batch that fills the off-peak hours with non-urgent work (re-embeddings, evals, light fine-tuning) instead of leaving the GPUs switched off or ticking over.
Reducing cold start so that scaling to zero and starting again is cheap and therefore viable as a utilisation strategy (see the sibling piece Speeding up cold start with Tensorizer).

The diagram: the €/token cascade by utilisation

The cost per token is not a number, it is a cascade that multiplies as utilisation falls:

Attributing the cost: LiteLLM virtual keys

With the €/token manufactured, it remains to attribute it to whoever consumed it. The piece that already sits in front of the engines, the L7 inference router, is also the natural place to count. LiteLLM Proxy makes chargeback real with four mechanisms, all documented:

Virtual keys with a budget. Each team or user receives a virtual key with a max_budget and a budget_duration. You can define several independent budget windows (for example one of 24 h and another of 30 d) that reset on their own cycle (Virtual Keys). When a team exhausts its monthly token budget, the proxy rejects with a budget exceeded error instead of carrying on spending GPU-hours nobody will be able to charge anywhere.

RPM/TPM rate limit. Each key carries requests per minute (RPM) and tokens per minute (TPM) limits (Budgets, Rate Limits). This is not just anti-abuse protection: it is the tool that stops a noisy tenant monopolising the cluster’s utilisation at everybody else’s expense. The rate limit is the capacity quota; the budget is the spend quota. They are orthogonal and you use both.

Automatic spend tracking. LiteLLM accounts for the spend of all known models by key, user and team, recording the API key, the user, the team_id, the request’s tags, the end user, the model group and the token counts (Spend Tracking). For the cluster’s self-hosted models, the cost per token is configured with the €/token we manufactured above, and that is the hook between the cost model and the real accounting.

Tags for cost centres. The tags let you track spend and set budgets per label, categorising costs by project, department or cost centre (Setting Tag Budgets). A tag is attached when the key is created and every request made with that key inherits the tag automatically; the proxy applies the label’s budget. That way chargeback maps cleanly onto the organisation’s accounting hierarchy without every team having to remember to tag by hand.

A declarative catalogue fragment, which fits into the litellm-config of the router post, materialising the manufactured cost and two tenants:

litellm_settings:
 # €/token manufactured in this post for the self-hosted model.
 # input/output in €/token (not per 1M); 0.186 €/1M = 1.86e-7 €/token at u=80%.
 model_cost_map:
 llama-70b-onprem:
 input_cost_per_token: 0.000000186
 output_cost_per_token: 0.000000186

# Virtual keys per team (via the /key/generate API or config):
# equipo-datos: max_budget=500 €/month, tpm_limit=200000, tags=["cc-datos"]
# equipo-soporte: max_budget=150 €/month, tpm_limit=60000, tags=["cc-soporte"]
# The proxy counts tokens, multiplies by model_cost_map, deducts from the budget,
# and attributes the spend to the tag → showback per cost centre, without touching the engine.

The complete attribution flow, from the request to the per-team report:

Showback vs chargeback: the distinction the FinOps Foundation insists on

The two words are constantly confused and the difference is one of accounting formality, not technology:

Showback — giving each team visibility of what it consumed and its cost, without really billing it to their budget. The report arrives, the team sees it, but the money does not move between cost centres.
Chargeback — really transferring the cost to the team’s or product’s budget, putting it in their P&L. The spend stops being a central IT cost and becomes an allocated cost.

The FinOps Foundation is explicit on two points worth carving in stone (Invoicing & Chargeback, Chargeback vs Showback). First: showback is a requirement of any FinOps practice; chargeback depends on the organisation’s accounting policy. Not all organisations can or want to move money between departments for GPU consumption; showback you can always do. Second, and counterintuitive: neither of the two is “more mature” than the other. The narrative that chargeback is the “grown-up” version of showback is false according to the framework itself. The practical recommendation: start with showback, giving visibility, then build cost allocation aligned to the organisational hierarchy, and only then, if the accounting policy supports it, switch chargeback on.

For the on-premise GPU cluster this means: the LiteLLM machinery (keys, tags, spend tracking) always produces the showback. Turning it into chargeback is a decision for finance, not for the platform team. The platform team guarantees that the numbers are correct and reproducible; who really pays is a governance decision.

Isolation: making the cost genuinely attributable

Chargeback is only honest if consumption is isolable. If two tenants share a GPU without partitioning and one saturates the HBM, the other suffers degradation it did not cause but which contaminates its cost attribution. Isolation has two planes.

Hard GPU isolation: MIG. Multi-Instance GPU physically partitions an H100 into up to 7 instances with dedicated memory and SMs; in a 7-way partition each instance has ~10 GB of HBM3 and its own SMs. MIG gives the strongest isolation: tenant A in its MIG instance cannot touch tenant B’s performance, and attribution is trivial because each instance is accountable separately (see Sharing a GPU for the detail of MIG vs time-slicing vs MPS). The cost: MIG instances are fixed, they are not resized hot, and if they sit empty they are fragmented and wasted capex, the coworking desk subdivided into booths nobody rents.

Logical Kubernetes isolation. On top of the cluster:

Namespaces per tenant — the boundary for RBAC, NetworkPolicy and quotas.
ResourceQuota — limits how many nvidia.com/gpu (or nvidia.com/mig-1g.10gb) a namespace can request. It is the GPU capacity quota at the scheduler level: the tenant cannot claim more GPUs than its quota grants, which bounds its maximum cost by construction.
PriorityClass — defines which workloads can evict others under pressure. The overnight re-embeddings batch runs with low priority and gives way to interactive inference; that way it fills the utilisation valleys without risking the paying tenant’s SLO. It is the piece that stops “raising utilisation with batch” from cannibalising quality of service.

The combination that works in the generic cluster: MIG to partition the GPUs between tenants that need hard isolation and clean attribution, ResourceQuota per namespace to bound each tenant’s cost, and PriorityClass so that filler work raises utilisation without touching the priority tenants.

Applied to the generic 4×H100 cluster

Let us bring it all down to the series’ cluster: 4 nodes × 4×H100 SXM 80 GB, 16 GPUs, ≈ 17.1 €/hour of fixed cost (≈ 150,000 €/year). Three teams share it: Data (production RAG, office-hours traffic), Support (ticket assistant, daytime peaks) and Platform (evals and re-embeddings batch, no urgency).

Splitting the cost. The cluster’s fixed cost (150,000 €/year) is split through LiteLLM’s spend tracking in proportion to the tokens consumed by each team, valued at the manufactured €/token. If in one month Data consumed 1,700 M tokens, Support 480 M and Platform 320 M, at 0.186 €/1M the showback is ≈ 316 € / 89 € / 60 €. The residue, the fixed cost of the GPUs nobody used because average utilisation was, say, 45 %, is the cost of idleness, and the governance decision is whether it is split between the tenants (penalises everybody equally) or charged to Platform as “cost of unsold capacity” (gives Platform an incentive to raise utilisation). The FinOps Foundation would say: make it visible first (showback of the cost of idleness), decide the split afterwards.

Quotas per tenant. Each team has its namespace with ResourceQuota: Data can claim up to 8 GPUs (2 replicas at TP=4), Support up to 4, Platform up to 4 but with a low PriorityClass, giving up its GPUs when Data or Support need them at peak. In LiteLLM, each team has its virtual key with a monthly max_budget and a tpm_limit that reflects its capacity quota.

MIG to isolate and attribute. For the small models (embeddings, reranker, a utility Llama 8B), an H100 split 7-way with MIG gives seven separately attributable instances: three for Data’s embeddings, two for Support’s reranker, two free for filler. Each instance is an independent accounting “desk”; the per-tenant showback comes straight from the scheduler.

The governance metric is utilisation, not the €/token. Here we close the circle. The KPI the cluster owner should look at daily is not “what the token costs”, that number only falls as a consequence, but DCGM_FI_DEV_GPU_UTIL, the DCGM exporter metric that indicates what fraction of the time the GPU is not idle (DCGM exporter). And with a critical nuance that already appeared in capacity planning: DCGM_FI_DEV_GPU_UTIL measures “not idle”, it does not measure useful work. A GPU can show 100 % GPU-util with the HBM saturated and low SM occupancy, producing few tokens. That is why serious FinOps crosses three signals: DCGM_FI_DEV_GPU_UTIL (is the GPU busy?), DCGM_FI_PROF_SM_OCCUPANCY and DCGM_FI_PROF_GR_ENGINE_ACTIVE (is it doing real work?), and vllm:gpu_cache_usage_perc (is the HBM well used?). The economic governance metric is useful tokens per GPU-hour, and it is watched from the GPU observability with DCGM dashboard.

An end-to-end assembly detail, LibreChat on top of LiteLLM with RAG, which closes the flow from tenant to cost, is covered in the (forthcoming) post on the end-to-end sovereign assistant; here it is enough to know that the metering point is always the proxy, never the engine.

Four traps of on-premise GPU FinOps

Trap 1 — comparing the on-premise €/token with the commercial API’s list price. The commercial API has margins, SLA and scale the on-premise cluster does not, but on-premise has the cost of idleness the API hides from you (they fill their GPUs with thousands of customers). The honest comparison is on-premise at its real utilisation vs the API at its real negotiated price, not the fantasy of on-premise at 100 %.

Trap 2 — forgetting the cost of idleness in the showback. If you only charge teams for the tokens they consumed, the cost of the idle GPUs disappears from the report and nobody sees it. That cost exists and somebody is paying it. Making it visible is the first step to reducing it.

Trap 3 — confusing high GPU-util with efficiency. A GPU at 100 % DCGM_FI_DEV_GPU_UTIL with the HBM saturated and low SM occupancy spends capex without producing proportional tokens. The goal is not “GPU at 100 %”, it is “useful tokens per euro”. Always cross util with SM occupancy and real throughput.

Trap 4 — chargeback before showback. Switching chargeback on, moving money, before the teams trust that the numbers are correct generates disputes that burn the whole FinOps programme. First visibility, then trust in the data, then, if the accounting policy allows it, the money.

References

NVIDIA — H100 Tensor Core GPU Datasheet (TDP 700 W, HBM3 3,35 TB/s): resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306.
LiteLLM — Virtual Keys (docs.litellm.ai/docs/proxy/virtual_keys), Budgets & Rate Limits (/docs/proxy/users), Spend Tracking (/docs/proxy/cost_tracking), Setting Tag Budgets (/docs/proxy/tag_budgets), Team Budgets (/docs/proxy/team_budgets).
FinOps Foundation — Invoicing & Chargeback Capability (finops.org/framework/capabilities/invoicing-chargeback/) y Data Analysis and Showback (finops.org/framework/previous-capabilities/analysis-showback/).
CloudZero — Chargeback vs. Showback: Cloud Cost Allocation Models Explained (cloudzero.com/blog/chargeback-vs-showback/).
NVIDIA — DCGM Exporter (docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_GR_ENGINE_ACTIVE.

Chargeback on lo0 — Tech Blog

GPU chargeback and showback in multi-tenancy: how to split cluster cost between teams

TL;DR

Showback vs chargeback: the FinOps Foundation definition

When to use each

GPU cost attribution with OpenCost

Allocation API: attribution parameters

Labelling inference pods

Shared costs and idle: the three modes

LiteLLM as the attribution point for token consumption

Virtual keys and teams

Configuring teams with a monthly budget

Endpoints for querying spend per team

Joining €/GPU-hour (iron) with tokens per team

Kueue as the GPU budget/quota mechanism

ClusterQueue, LocalQueue and cohorts

Quota concepts

Example YAML: 3 teams in the llm-platform cohort

How Kueue materialises the “GPU budget”

Cost model: an example with 4×H100 SXM and 3 teams

Node price

Monthly scenario: 3 teams, heterogeneous utilisation

Token consumption per team (LiteLLM spend logs)

Monthly chargeback report (showback + tokens)

Attribution dimensions × policy table

End-to-end data flow

Fair sharing and preemption: how Kueue reclaims quota

Configuring OpenCost for the node price

Reference PromQL for chargeback dashboards

Sources

See also

From GPU-hour to cost per token: the metric that compares on-prem and cloud

What this article covers

The identity: from GPU-hour to token

The two halves of cost per token

The gateway: LiteLLM in depth

The self-hosted trap: LiteLLM does not know what your own token costs you

Configuring custom pricing: an example

Cost per token changes with throughput: read it off the sweep

Cost per request: not only per token

Cost per model and per tier

Attribution: from cost to team and product

A combined report example

Governing spend: budgets, caps and alerts

On-prem vs European cloud comparison (the reason for all of it)

From cost per token to product price

Detailed comparison: on-prem, European cloud and API

Prompt caching and effective cost

State of the art 2026

Limits and traps (data-driven)

Closing

See also

Sources

FinOps and multi-tenancy on the GPU cluster: who pays for what

TL;DR

The problem the cloud hides from you and on-premise forces you to solve

The analogy: the coworking space with metered services

The token as the unit of cost

Cost model of a GPU: from capex to €/token

Depreciation (spread capex)

Energy (power × PUE × price_kWh)

The total €/GPU-hour

From €/GPU-hour to €/token

Utilisation is the lever (with numbers)

The diagram: the €/token cascade by utilisation

Attributing the cost: LiteLLM virtual keys

Showback vs chargeback: the distinction the FinOps Foundation insists on

Isolation: making the cost genuinely attributable

Applied to the generic 4×H100 cluster

Four traps of on-premise GPU FinOps

See also

References