FinOps and multi-tenancy on the GPU cluster: who pays for what

Third instalment in an operational series on how to squeeze a generic on-premise LLM inference cluster of 4×H100 SXM 80 GB. The sibling pieces are Multimodal VLM on-premise with vLLM, serving vision-language models on the same cluster, and Speeding up cold start with Tensorizer, cutting the model load time from disk to HBM. This post closes the economic question the other two leave open: once the cluster serves several models for several teams, who pays for what?

TL;DR

An on-premise GPU cluster does not have the property that makes FinOps easy in the cloud: no monthly invoice arrives telling you what each compute hour cost. The cost has to be manufactured from capex and energy. The right unit is the token, and it is derived in two steps. First the €/GPU-hour: capex depreciation spread over the hours of useful life, plus energy (power × PUE × price_kWh). For an H100 SXM with an assumed capex of 30,000 € and 4 years of life at 90 % availability, that gives ≈ 1.07 €/GPU-hour (0.95 of depreciation + 0.12 of energy at 0.12 €/kWh and PUE 1.4). Second, the €/1M tokens: divide that €/GPU-hour by the sustained throughput. At 2,000 useful tok/s, ≈ 0.148 €/1M tokens; but that number only holds if the GPU is at 100 % useful utilisation. At 20 % utilisation the same token costs 4× more (0.74 €/1M), because capex and energy keep being paid even if the GPU is idle. That is the central thesis: the FinOps goal is not to negotiate the price of the token down, there is no provider to negotiate with, but to raise useful utilisation, which is the only term that moves cost by an order of magnitude. Attribution (chargeback / showback) is made real with LiteLLM virtual keys: per-team and per-user budget, RPM/TPM rate limit, automatic spend tracking and tags to assign spend to cost centres. Isolation between tenants is achieved with MIG (hard GPU partitioning), namespaces, ResourceQuota and PriorityClass. The FinOps Foundation rule worth carving in stone: showback always, chargeback only if the organisation’s accounting policy supports it; neither of the two is “more mature”. And the governance metric of the shared cluster is not the €/token but DCGM_FI_DEV_GPU_UTIL.

The problem the cloud hides from you and on-premise forces you to solve

In the cloud, the cost of a GPU is an observable fact: AWS, GCP or Azure charge you per GPU-hour and the invoice arrives itemised. FinOps in that world consists of reading an invoice that already exists and attributing it. On-premise there is no such invoice. The GPU was bought once, eighteen months ago, on a purchase order nobody remembers; the electricity bill arrives for the whole building with no breakdown of what fraction was the cluster’s; and the datacenter PUE is a number the facilities team knows but the AI team never asked about.

The first job of on-premise FinOps, then, is not to attribute a cost but to manufacture one. You have to build, from explicit assumptions, the equivalent of the “price per GPU-hour” that in the cloud is given to you. And once manufactured, you have to understand that this number is not a property of the hardware, like VRAM or TDP, but a function of how the hardware is used. An idle H100 costs exactly the same as an H100 at 100 %: the capex is already paid and idle energy is not zero. The only thing that changes is how many useful tokens that fixed cost produced. That is where the whole thesis of this post comes from.

The analogy: the coworking space with metered services

A coworking space manages a finite physical space and rents it to teams. The cost model has exactly the structure of the GPU cluster, and the analogy holds down to the last detail.

The desk-hour is the spread capex. The manager paid the rent on the premises, the furniture and the refurbishment, a one-off outlay or a fixed monthly cost, and spreads it over the available desk-hours in the month. Each desk has a base cost that does not depend on whether anybody uses it: the walls, the table and the chair cost the same empty as occupied. This is the depreciation of the GPU: capex divided by the hours of useful life.

The metered kWh is the energy. On top of the desk-hour, the coworking space measures the actual electricity consumption, the plugged-in laptop, the screen, the air conditioning in that room, and bills it separately. This is the energy of the GPU: power × hours × price_kWh, multiplied by the datacenter’s PUE, which is the coworking space also charging you for the fraction of the building’s air conditioning that cools your room.

The empty desk is paid for by whoever booked it. Here is the heart of the matter. If a team books ten desks for the whole month and only uses two, it pays for all ten anyway. The empty desk still takes up space the manager cannot resell, still depreciates furniture, still counts as committed capacity. In the cluster: a GPU assigned to a tenant that uses it at 20 % costs the same as if it used it at 100 %, and the idle 80 % is wasted capex that somebody is paying for.

The manager optimises occupancy, not tariff. A competent coworking manager does not obsess over lowering the price of the desk-hour, it is a sunk cost, the premises are already paid for. They obsess over the occupancy rate: every empty desk is margin lost forever, because an unsold desk-hour cannot be stored. The FinOps equivalent: the goal is not to lower the €/token, it is to raise utilisation, because the €/token falls only as a consequence of utilisation rising. There is no provider to squeeze; the only margin lies in not leaving GPUs idle.

The analogy has an honest limit worth naming: in a coworking space, two people cannot physically share the same chair. On a GPU you can, with time-slicing, MPS or MIG (see Sharing a GPU), and that is precisely the technical lever that lets you raise occupancy beyond what the physical coworking space would allow. But the economics of fixed cost + metered cost is identical.

The token as the unit of cost

Why the token and not the GPU-hour, or the request, or the model? Because the token is the only unit comparable across heterogeneous workloads. A GPU-hour of Llama 8B and one of Llama 70B produce radically different amounts of “useful work”; a RAG request with a 50-token answer and one generating a 3,000-token report are not comparable. The token, specifically the pair (input_tokens, output_tokens), normalises all of that. It is the unit LiteLLM accounts for natively, the one that appears in the OTel attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (see LLM tracing with OpenTelemetry GenAI), and the one commercial providers already use to charge. Adopting the token as the internal currency makes on-premise chargeback directly comparable with the cloud alternative, which is exactly the comparison management wants to see.

The two derived metrics that matter:

• Tokens/€ — how many useful tokens each euro of total cluster cost buys. It rises when utilisation rises. It is the metric of economic efficiency.
• Tokens/W — how many tokens each watt consumed produces. It is the metric of energy efficiency and, in a datacenter with limited power (the usual case on-premise), it is often the real hard constraint: you cannot add more GPUs because there are no more kW in the rack, so every watt has to perform.

Cost model of a GPU: from capex to €/token

The hourly cost of a GPU has two addends. Let us put all the assumptions in writing, because, as in capacity planning, a cost calculation without written assumptions is a disposable calculation.

Depreciation (spread capex)

where the hours of useful life are the depreciation years × 8,760 h/year × the effective availability (the GPU is not available 100 % of the time: there is maintenance, restarts, update windows). Explicit, generic assumptions (they are not figures from any provider or any real purchase):

• Capex per GPU: 30,000 € — a reasonable assumption that includes not just the GPU but its pro-rata share of the server, power supply, cooling, NVLink/InfiniBand network and rack. A “bare” H100 SXM costs less, but honest FinOps spreads the cost of the whole node across its GPUs.
• Accounting useful life: 4 years. That is aggressive-realistic for a datacenter GPU; some depreciate over 3, others over 5.
• Effective availability: 90 %.

Energy (power × PUE × price_kWh)

Assumptions:

• GPU power: the TDP of the H100 SXM is 700 W according to the NVIDIA datasheet (H100 datasheet). Under sustained inference load it hovers around that value; we use 0.7 kW as the average power under load. (At idle it drops a lot, 60–100 W, but the FinOps of reserved capacity reasons about power under load, which is what limits the rack.)
• PUE (Power Usage Effectiveness): how many watts enter the datacenter for every watt that reaches the chip. A modest, well-managed on-premise datacenter sits at 1.4; a bad one at 2.0; the hyperscalers boast of 1.1. We use 1.4.
• Price per kWh: 0.12 €/kWh — a generic assumption for an industrial tariff; it varies enormously by country and contract.

The total €/GPU-hour

Two important readings of this number. First: depreciation dominates (89 % of the cost); energy is 11 %. This inverts many people’s intuition, who believe “the expensive part is the electricity”. With these assumptions, the expensive part is having bought the GPU, and that is why leaving it idle hurts so much. Second: the number is per-GPU; for the 4×H100 node it is ≈ 4.28 €/hour, and for the complete generic cluster of 4 nodes (16 GPUs), ≈ 17.1 €/hour ≈ 150,000 €/year of fixed cost that is paid whether it is used or not.

From €/GPU-hour to €/token

Now we divide the hourly cost by the tokens the GPU produces in an hour. Here sustained throughput comes in. Let us suppose, an order-of-magnitude figure, checkable with vllm bench serve, that a Llama 70B replica on 4×H100 (TP=4) sustains 2,000 useful tokens/s aggregated under good concurrency.

The cost of that replica (4 GPUs) is $4 \times 1.07 = 4.28$ €/hour. Therefore:

Per individual GPU, if we take a smaller replica (e.g. Llama 8B FP8 on 1 GPU at ~2,000 tok/s):

These numbers are the theoretical floor at 100 % utilisation. Nobody operates at 100 %. And that is the whole story.

Utilisation is the lever (with numbers)

The €/token in the previous section assumes the GPU produces 7.2 M tokens every hour, hour after hour. In practice it produces that only during peak hours. The rest of the time it is partly idle: at night, at weekends, between traffic peaks. The useful utilisation $u$ is the fraction of the theoretical capacity that really turns into billable tokens.

The hourly cost is fixed (1.07 €/GPU-hour is paid come rain or shine), but the tokens produced scale with $u$:

The requested contrast, worked out:

Direct reading: going from 20 % to 80 % utilisation divides the cost per token by four ($0.744 \to 0.186$). No tariff negotiation, no quantization, no hardware change gives you that factor of 4 so cheaply. Quantizing from BF16 to FP8 can double throughput, and therefore halves the €/token, but it degrades quality and demands evals (see Quantization for inference); raising utilisation from 20 to 80 % does not touch the quality of a single token.

This reformulates GPU cluster FinOps in a single sentence: the €/token is not something you negotiate, it is something you earn by filling the GPUs. And raising utilisation is an engineering problem that already has its pieces:

• Autoscaling that switches replicas off when traffic drops, so as not to pay for idle GPU (see LLM autoscaling on Kubernetes with KEDA). Switching off a 4-GPU replica for eight hours overnight saves ≈ 34 € a day of real cost; multiplied by replicas and by days, that is the difference between a profitable cluster and one that bleeds.
• Sharing a GPU between small workloads with MIG or time-slicing, so that two tenants that individually would use 30 % together fill a GPU to 60 % (see Sharing a GPU).
• Overnight batch that fills the off-peak hours with non-urgent work (re-embeddings, evals, light fine-tuning) instead of leaving the GPUs switched off or ticking over.
• Reducing cold start so that scaling to zero and starting again is cheap and therefore viable as a utilisation strategy (see the sibling piece Speeding up cold start with Tensorizer).

The diagram: the €/token cascade by utilisation

The cost per token is not a number, it is a cascade that multiplies as utilisation falls:

Attributing the cost: LiteLLM virtual keys

With the €/token manufactured, it remains to attribute it to whoever consumed it. The piece that already sits in front of the engines, the L7 inference router, is also the natural place to count. LiteLLM Proxy makes chargeback real with four mechanisms, all documented:

Virtual keys with a budget. Each team or user receives a virtual key with a max_budget and a budget_duration. You can define several independent budget windows (for example one of 24 h and another of 30 d) that reset on their own cycle (Virtual Keys). When a team exhausts its monthly token budget, the proxy rejects with a budget exceeded error instead of carrying on spending GPU-hours nobody will be able to charge anywhere.

RPM/TPM rate limit. Each key carries requests per minute (RPM) and tokens per minute (TPM) limits (Budgets, Rate Limits). This is not just anti-abuse protection: it is the tool that stops a noisy tenant monopolising the cluster’s utilisation at everybody else’s expense. The rate limit is the capacity quota; the budget is the spend quota. They are orthogonal and you use both.

Automatic spend tracking. LiteLLM accounts for the spend of all known models by key, user and team, recording the API key, the user, the team_id, the request’s tags, the end user, the model group and the token counts (Spend Tracking). For the cluster’s self-hosted models, the cost per token is configured with the €/token we manufactured above, and that is the hook between the cost model and the real accounting.

Tags for cost centres. The tags let you track spend and set budgets per label, categorising costs by project, department or cost centre (Setting Tag Budgets). A tag is attached when the key is created and every request made with that key inherits the tag automatically; the proxy applies the label’s budget. That way chargeback maps cleanly onto the organisation’s accounting hierarchy without every team having to remember to tag by hand.

A declarative catalogue fragment, which fits into the litellm-config of the router post, materialising the manufactured cost and two tenants:

litellm_settings:
  # €/token manufactured in this post for the self-hosted model.
  # input/output in €/token (not per 1M); 0.186 €/1M = 1.86e-7 €/token at u=80%.
  model_cost_map:
    llama-70b-onprem:
      input_cost_per_token: 0.000000186
      output_cost_per_token: 0.000000186

# Virtual keys per team (via the /key/generate API or config):
#   equipo-datos:   max_budget=500 €/month, tpm_limit=200000, tags=["cc-datos"]
#   equipo-soporte: max_budget=150 €/month, tpm_limit=60000,  tags=["cc-soporte"]
# The proxy counts tokens, multiplies by model_cost_map, deducts from the budget,
# and attributes the spend to the tag → showback per cost centre, without touching the engine.

The complete attribution flow, from the request to the per-team report:

Showback vs chargeback: the distinction the FinOps Foundation insists on

The two words are constantly confused and the difference is one of accounting formality, not technology:

• Showback — giving each team visibility of what it consumed and its cost, without really billing it to their budget. The report arrives, the team sees it, but the money does not move between cost centres.
• Chargeback — really transferring the cost to the team’s or product’s budget, putting it in their P&L. The spend stops being a central IT cost and becomes an allocated cost.

The FinOps Foundation is explicit on two points worth carving in stone (Invoicing & Chargeback, Chargeback vs Showback). First: showback is a requirement of any FinOps practice; chargeback depends on the organisation’s accounting policy. Not all organisations can or want to move money between departments for GPU consumption; showback you can always do. Second, and counterintuitive: neither of the two is “more mature” than the other. The narrative that chargeback is the “grown-up” version of showback is false according to the framework itself. The practical recommendation: start with showback, giving visibility, then build cost allocation aligned to the organisational hierarchy, and only then, if the accounting policy supports it, switch chargeback on.

For the on-premise GPU cluster this means: the LiteLLM machinery (keys, tags, spend tracking) always produces the showback. Turning it into chargeback is a decision for finance, not for the platform team. The platform team guarantees that the numbers are correct and reproducible; who really pays is a governance decision.

Isolation: making the cost genuinely attributable

Chargeback is only honest if consumption is isolable. If two tenants share a GPU without partitioning and one saturates the HBM, the other suffers degradation it did not cause but which contaminates its cost attribution. Isolation has two planes.

Hard GPU isolation: MIG. Multi-Instance GPU physically partitions an H100 into up to 7 instances with dedicated memory and SMs; in a 7-way partition each instance has ~10 GB of HBM3 and its own SMs. MIG gives the strongest isolation: tenant A in its MIG instance cannot touch tenant B’s performance, and attribution is trivial because each instance is accountable separately (see Sharing a GPU for the detail of MIG vs time-slicing vs MPS). The cost: MIG instances are fixed, they are not resized hot, and if they sit empty they are fragmented and wasted capex, the coworking desk subdivided into booths nobody rents.

Logical Kubernetes isolation. On top of the cluster:

• Namespaces per tenant — the boundary for RBAC, NetworkPolicy and quotas.
• ResourceQuota — limits how many nvidia.com/gpu (or nvidia.com/mig-1g.10gb) a namespace can request. It is the GPU capacity quota at the scheduler level: the tenant cannot claim more GPUs than its quota grants, which bounds its maximum cost by construction.
• PriorityClass — defines which workloads can evict others under pressure. The overnight re-embeddings batch runs with low priority and gives way to interactive inference; that way it fills the utilisation valleys without risking the paying tenant’s SLO. It is the piece that stops “raising utilisation with batch” from cannibalising quality of service.

The combination that works in the generic cluster: MIG to partition the GPUs between tenants that need hard isolation and clean attribution, ResourceQuota per namespace to bound each tenant’s cost, and PriorityClass so that filler work raises utilisation without touching the priority tenants.

Applied to the generic 4×H100 cluster

Let us bring it all down to the series’ cluster: 4 nodes × 4×H100 SXM 80 GB, 16 GPUs, ≈ 17.1 €/hour of fixed cost (≈ 150,000 €/year). Three teams share it: Data (production RAG, office-hours traffic), Support (ticket assistant, daytime peaks) and Platform (evals and re-embeddings batch, no urgency).

Splitting the cost. The cluster’s fixed cost (150,000 €/year) is split through LiteLLM’s spend tracking in proportion to the tokens consumed by each team, valued at the manufactured €/token. If in one month Data consumed 1,700 M tokens, Support 480 M and Platform 320 M, at 0.186 €/1M the showback is ≈ 316 € / 89 € / 60 €. The residue, the fixed cost of the GPUs nobody used because average utilisation was, say, 45 %, is the cost of idleness, and the governance decision is whether it is split between the tenants (penalises everybody equally) or charged to Platform as “cost of unsold capacity” (gives Platform an incentive to raise utilisation). The FinOps Foundation would say: make it visible first (showback of the cost of idleness), decide the split afterwards.

Quotas per tenant. Each team has its namespace with ResourceQuota: Data can claim up to 8 GPUs (2 replicas at TP=4), Support up to 4, Platform up to 4 but with a low PriorityClass, giving up its GPUs when Data or Support need them at peak. In LiteLLM, each team has its virtual key with a monthly max_budget and a tpm_limit that reflects its capacity quota.

MIG to isolate and attribute. For the small models (embeddings, reranker, a utility Llama 8B), an H100 split 7-way with MIG gives seven separately attributable instances: three for Data’s embeddings, two for Support’s reranker, two free for filler. Each instance is an independent accounting “desk”; the per-tenant showback comes straight from the scheduler.

The governance metric is utilisation, not the €/token. Here we close the circle. The KPI the cluster owner should look at daily is not “what the token costs”, that number only falls as a consequence, but DCGM_FI_DEV_GPU_UTIL, the DCGM exporter metric that indicates what fraction of the time the GPU is not idle (DCGM exporter). And with a critical nuance that already appeared in capacity planning: DCGM_FI_DEV_GPU_UTIL measures “not idle”, it does not measure useful work. A GPU can show 100 % GPU-util with the HBM saturated and low SM occupancy, producing few tokens. That is why serious FinOps crosses three signals: DCGM_FI_DEV_GPU_UTIL (is the GPU busy?), DCGM_FI_PROF_SM_OCCUPANCY and DCGM_FI_PROF_GR_ENGINE_ACTIVE (is it doing real work?), and vllm:gpu_cache_usage_perc (is the HBM well used?). The economic governance metric is useful tokens per GPU-hour, and it is watched from the GPU observability with DCGM dashboard.

An end-to-end assembly detail, LibreChat on top of LiteLLM with RAG, which closes the flow from tenant to cost, is covered in the (forthcoming) post on the end-to-end sovereign assistant; here it is enough to know that the metering point is always the proxy, never the engine.

Four traps of on-premise GPU FinOps

Trap 1 — comparing the on-premise €/token with the commercial API’s list price. The commercial API has margins, SLA and scale the on-premise cluster does not, but on-premise has the cost of idleness the API hides from you (they fill their GPUs with thousands of customers). The honest comparison is on-premise at its real utilisation vs the API at its real negotiated price, not the fantasy of on-premise at 100 %.

Trap 2 — forgetting the cost of idleness in the showback. If you only charge teams for the tokens they consumed, the cost of the idle GPUs disappears from the report and nobody sees it. That cost exists and somebody is paying it. Making it visible is the first step to reducing it.

Trap 3 — confusing high GPU-util with efficiency. A GPU at 100 % DCGM_FI_DEV_GPU_UTIL with the HBM saturated and low SM occupancy spends capex without producing proportional tokens. The goal is not “GPU at 100 %”, it is “useful tokens per euro”. Always cross util with SM occupancy and real throughput.

Trap 4 — chargeback before showback. Switching chargeback on, moving money, before the teams trust that the numbers are correct generates disputes that burn the whole FinOps programme. First visibility, then trust in the data, then, if the accounting policy allows it, the money.

See also

• Multimodal VLM on-premise with vLLM — sibling piece: serving vision-language models on the same shared cluster whose cost we attribute here.
• Speeding up cold start with Tensorizer — sibling piece: cutting the model load time makes scaling to zero viable, which is the most direct utilisation lever.
• Capacity planning for on-premise LLM inference — the sustained throughput the €/token comes from is computed there; FinOps monetises it.
• Sharing a GPU: time-slicing, MPS and MIG — the partitioning that lets you isolate tenants and raise utilisation by combining small workloads.
• The LLM inference router: the L7 switchboard — where LiteLLM lives and where every token is counted and attributed.
• LLM autoscaling on Kubernetes with KEDA — switching off idle replicas is the number one lever to raise useful utilisation and lower the €/token.
• GPU observability for LLM inference with DCGM — where DCGM_FI_DEV_GPU_UTIL comes from, the governance metric of the shared cluster.
• LiteLLM and Langfuse: the operational pair — where the prices computed here get registered in LiteLLM, and why a model with no price silently produces zero spend and budgets that never run out.
• Virtual keys, budgets and limits in LiteLLM — how the budgets computed here are enforced in the proxy, and the condition in the code by which a team key overrides its owner’s personal cap.

References

• NVIDIA — H100 Tensor Core GPU Datasheet (TDP 700 W, HBM3 3,35 TB/s): resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306.
• LiteLLM — Virtual Keys (docs.litellm.ai/docs/proxy/virtual_keys), Budgets & Rate Limits (/docs/proxy/users), Spend Tracking (/docs/proxy/cost_tracking), Setting Tag Budgets (/docs/proxy/tag_budgets), Team Budgets (/docs/proxy/team_budgets).
• FinOps Foundation — Invoicing & Chargeback Capability (finops.org/framework/capabilities/invoicing-chargeback/) y Data Analysis and Showback (finops.org/framework/previous-capabilities/analysis-showback/).
• CloudZero — Chargeback vs. Showback: Cloud Cost Allocation Models Explained (cloudzero.com/blog/chargeback-vs-showback/).
• NVIDIA — DCGM Exporter (docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_GR_ENGINE_ACTIVE.