---
title: "From GPU-hour to cost per token: the metric that compares on-prem and cloud"
url: https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/
markdown: https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/index.md
lang: en
date: 2026-06-14
lastmod: 2026-09-13
author: "José Manuel Román"
description: "A data deep dive: how the cost per token and per request of your own inference is calculated, why it is not a list price but €/GPU-hour divided by throughput, how to instrument it with the LiteLLM gateway (spend logs, response_cost, custom pricing, per-team attribution) and how to compare on-prem against European cloud. With sources."
tags: ["finops","cost-per-token","litellm","gateway","opencost","chargeback","sovereignty","data-series"]
translations:
  es: https://blog.lo0.es/posts/coste-por-token-y-por-request/index.md
---

# From GPU-hour to cost per token: the metric that compares on-prem and cloud


> Notation: amounts in **euros (N €)**, decimals with a point. The dollar sign is not used
> (on this site it is a formula delimiter).

## What this article covers

Fourth article in the FinOps track (A4), and the one that closes out the metric that gives
the whole pillar its meaning: **cost per token**. In A2 we saw how OpenCost assigns the cost of
the iron (the pod, in €/hour); in A3, which tool to use. But the cost of the pod is not the
metric that compares on-prem with cloud, nor the one the business understands: that one is
**cost per token** (or per request). This article explains how it is really calculated (spoiler:
for your own inference it **is not a list price, it is €/GPU-hour divided by throughput**), how
to instrument it with a gateway (LiteLLM), how to attribute it by team and product, and how
to compare it against European cloud. No recommendations; only the mechanics and the numbers.

---

## The identity: from GPU-hour to token

Cost per million tokens (CPM) is the cost of the iron divided by what it produces:

$$\text{cost per token} = \frac{\text{cost of the iron (€/h)}}{\text{throughput (tok/s)} \times 3600 / 10^6}$$

This is the central formula of the FinOps track, and it has a consequence many people never quite
internalise: for a self-hosted model, the cost of a token **does not exist as a fixed
price**. It depends on two things that change: the cost of the iron (from OpenCost, in
€/hour) and the throughput (from your benchmark, in tok/s). The same model on the same node
costs half as much per token if you double the throughput with an optimisation, because the
denominator changes, not the numerator. The cost per token of your own inference is **a function
of your efficiency**, not a tariff.

That sets it radically apart from an external API, where the price per token really is a
fixed tariff the provider charges you. Comparing the two requires turning your variable cost
into a number, and that is where the iron (A2) meets the tokens (this article).

---

## The two halves of cost per token

Cost per token comes from joining two measurements that live in different places:

> **Diagram:** The two halves of cost per token: the cost of the iron from OpenCost and the token count from the gateway, which divided give the cost per token — [see the HTML version](https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/)

The iron half comes from OpenCost (article A2): the by-usage cost of the vLLM pod, in
€/hour. The token half comes from the gateway, which intercepts every request and counts
tokens per model and per team. Divide, and out comes the cost per token and, crucially, **per
team**, which is what enables chargeback.

---

## The gateway: LiteLLM in depth

The gateway sits between the application and the engine (vLLM) or the provider, and intercepts
every request. LiteLLM is the OSS reference: it proxies over 100 models and **tracks spend
by keys, users and teams**, logging spend for all known models automatically
([LiteLLM · Spend Tracking](https://docs.litellm.ai/docs/proxy/cost_tracking)).

Its key mechanisms:

| Mechanism | What it does |
|---|---|
| `response_cost` | the cost of each response, available in logging (`kwargs["response_cost"]`) |
| **Spend logs** | one row per request with tokens, model, user, key and cost (SQLite/PostgreSQL) |
| **Tags** | tag each request by team, project, environment and model *tier* |
| **Budgets** | budgets and rate limits per team or user |
| **Custom callbacks** | bespoke loggers for key/user/model/tokens/cost |

With the spend logs table (via the database adapter) you have the detail of every request to
build cost dashboards ([LiteLLM · Logging](https://docs.litellm.ai/docs/proxy/logging)), and you
can tag each request with team/project/environment for per-team analysis, with notifications
(Slack) and hard caps for keys that must not exceed a limit. In practice it is the platform's
token meter and cost allocator.

---

## The self-hosted trap: LiteLLM does not know what your own token costs you

Here is the point almost nobody wires up correctly. LiteLLM tracks cost using list prices
for known models (the API ones: OpenAI, Anthropic and so on). But for a model self-hosted on
your vLLM there **is no list price**, because you pay nobody per token. If you tell it
nothing, LiteLLM does not know the real cost of your own token. The solution is custom
pricing: declaring to LiteLLM the cost per token of your self-hosted model
([LiteLLM · Custom Pricing](https://docs.litellm.ai/docs/proxy/custom_pricing)).

And where does that number come from? From the identity above: the cost of the iron
(OpenCost) divided by the throughput (your benchmark). That is, the correct flow is:

1. OpenCost gives you the cost of the vLLM pod: ~11 €/hour (A2).
2. Your benchmark gives you the throughput: ~2,800 tok/s (track B).
3. The identity gives the cost per token: 11 ÷ (2,800 × 3,600 / 10⁶) ≈ **1.09 €/1M tok**.
4. That 1.09 €/1M goes into LiteLLM as custom pricing.

Only then does the `response_cost` of each request reflect the real cost of your inference,
rather than a zero or somebody else's list price. Without this step, your LiteLLM cost dashboards
lie about your own models. It is the explicit junction of the three tracks: iron (FinOps),
throughput (benchmarking) and measurement (gateway).

---

### Configuring custom pricing: an example

In LiteLLM, the real cost of a self-hosted model is declared in its configuration. An example
with the cost per token derived from the identity (1.09 €/1M ÷ 10⁶ = 0.00000109 €/token):

```yaml
# LiteLLM config.yaml — real cost of a self-hosted model
model_list:
  - model_name: llama-3.1-70b-onprem
    litellm_params:
      model: openai/llama-3.1-70b           # your vLLM endpoint
      api_base: http://vllm:8000/v1
      input_cost_per_token:  0.00000109      # €/token (from OpenCost ÷ throughput)
      output_cost_per_token: 0.00000109      # refine if you separate prefill/decode
```

With that, every `response_cost` LiteLLM records reflects the real cost of your iron. The
operational note: this number must be updated when the cost of the node changes (new
depreciation, a different energy price) or the throughput does (an engine optimisation). It is
not a constant; it is the output of OpenCost and of the benchmark, and it is recalculated
whenever either of the two changes. Automating that recalculation, from node cost and measured
throughput to custom pricing, is what keeps cost dashboards honest.

---

## Cost per token changes with throughput: read it off the sweep

Because throughput is the denominator, cost per token is not a point, it is a curve over the
concurrency sweep. On the ~11 €/hour node:

| Concurrency | Throughput (tok/s) | Cost per 1M tokens |
|---|---|---|
| 1 | 350 | **~8.73 €** |
| 8 | 2,100 | **~1.46 €** |
| 16 | 3,330 (goodput) | **~0.92 €** |
| 24 | 3,900 raw / 2,420 goodput | ~0.78 € raw / **~1.26 € on goodput** |

Important readings: at low concurrency the cost per token is sky-high (the fixed cost of the
node is shared among few tokens); cost falls as occupancy rises. But watch the last row: if you
compute cost on raw throughput (3,900) you get 0.78 €, whereas if you compute it on
goodput (2,420, the part that meets the SLO) you get 1.26 €, and the latter is the real
cost, because requests that violate the SLO do not count as useful production. An honest cost
per token is computed on **goodput**, not on catalogue throughput. This is where the cost track
meets the benchmarking one: the correct denominator is goodput.

---

## Cost per request: not only per token

Cost per token is the unit; cost per request is what a complete request bills. A request
consumes **input tokens (prompt) plus output tokens (generation)**, and both count:

$$\text{cost per request} = (\text{input tokens} + \text{output tokens}) \times \text{cost per token}$$

With nuances: in many engines the prefill (input) and the decode (output) have different
costs per token (decode is memory-bound and more expensive per token), so a fine-grained cost
model distinguishes the two. For most cases a weighted average cost per token will do; to
optimise, it pays to separate prefill and decode. LiteLLM records input and output tokens
separately in its spend logs, so the data is there to refine with.

Cost per request is what connects to the product: if a feature consumes 1,500 tokens on
average (1,200 of prompt plus 300 of output) and the cost is 1.09 €/1M, each use costs
~0.0016 €, the basis for setting margin, as seen in the FinOps introduction.

---

## Cost per model and per tier

A platform does not serve a single model. Cost per token varies by model and by
configuration, and the gateway tells them apart. An 8B and a 70B on the same node have very
different costs per token: the 8B yields far more throughput (more tokens/s) for the same
GPU-hour, so its cost per token is a fraction of the 70B's. The design consequence: routing
each request to the cheapest model that meets the quality bar is a direct cost lever, since
serving a 70B where an 8B suffices means overpaying per token.

| Model (example, same node) | Relative throughput | Relative cost per token |
|---|---|---|
| 8B | high | low |
| 70B FP8 | medium | medium |
| 70B FP16 | low | high |

The gateway, with its per-model custom pricing, makes this difference visible and enables
**cost-based routing**: classify the request and send it to the appropriate *tier*. It connects
with the L7 router of the serving track: the gateway not only measures cost, it can decide
on the basis of it. And quantisation (FP16→FP8) shows up again as a lever: the same model in FP8
raises throughput and lowers cost per token, moving the figure without changing model.

---

## Attribution: from cost to team and product

The reason for putting a gateway in place is not only to measure total cost, it is to share it
out. By tagging each request with team, product and environment, LiteLLM builds the split by
business dimension:

> **Diagram:** Cost per token attribution: the gateway tags each request by team and product, aggregates spend and produces chargeback with budgets and caps — [see the HTML version](https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/)

With per-team budgets and **hard caps** per key, the gateway does not only report: it
governs. A team that goes over budget gets an alert; a key with a hard cap cannot spend any
more. It is the FinOps *Operate* phase applied to the token.

### A combined report example

Joining the iron (OpenCost) and the tokens (gateway), the monthly cost-per-token-per-team report
on the example node:

| Team | Tokens/month | Applied cost per token | Charged cost |
|---|---|---|---|
| A · chat-prod | 240M | 1.09 €/1M (70B FP8) | ~262 € |
| B · batch | 90M | 0.40 €/1M (8B) | ~36 € |
| C · experimentation | 15M | 1.09 €/1M (70B) | ~16 € + idle |

What it reveals: A consumes a lot but at an efficient cost (FP8, high utilisation); B uses a
cheap model (8B) for its workload; C consumes little but, as we saw in A2, drags along the idle
cost of its underused GPU. The token report plus the OpenCost iron report give the full
picture: who spends, on which model, with what efficiency. It is the chargeback that neither
half would produce alone. And the per-team figure is what gets taken into the budget
conversation: not "the cluster costs X", but "your team consumed Y million tokens at Z €/million".

---

## Governing spend: budgets, caps and alerts

Measuring without governing is a dashboard; the value lies in closing the loop. LiteLLM allows
three levels of control over spend:

| Control | What it does | When |
|---|---|---|
| **Soft budget** | alert (Slack) when a threshold is crossed | tracking teams |
| **Rate limit** | limits requests/tokens per minute | avoiding cost spikes |
| **Hard cap** | the key cannot spend any more | keys that must never overrun |

The healthy pattern: soft budgets with alerts for the teams (so they adjust their consumption),
rate limits to smooth spikes, and hard caps only on critical keys (a service that must not run
away). With this, cost per token stops being an after-the-fact figure and becomes a real-time
control: the FinOps *Operate* phase, applied to inference.

---

## On-prem vs European cloud comparison (the reason for all of it)

Cost per token exists to answer one question: is it cheaper to serve on my own iron or to
rent? With the identity and European figures:

| Option | Cost per 1M tokens | Notes |
|---|---|---|
| Depreciated on-prem (8×H100, 2,800 tok/s) | **~1.09 €** | your iron, your sovereignty |
| European cloud Scaleway (8×H100, on-demand) | **~2.17 €** | EU, no iron to operate |
| External API (proprietary model) | provider tariff | no data sovereignty |

Cost per token is what makes the incomparable comparable: your variable cost (iron ÷
throughput) against a provider's fixed tariff. And it exposes the lever: if an optimisation
raises your throughput from 2,800 to 4,200 tok/s, your on-prem cost drops to ~0.73 €/1M,
widening the advantage over renting. That is why benchmarking (track B) is, indirectly, the tool
that moves cost per token the most: every goodput improvement translates into this number.

---

## From cost per token to product price

Cost per token closes the loop with the business when it becomes cost per unit of product.
The chain: cost per token (from the identity) × tokens per use of a feature = marginal cost of
that feature. With that, three business decisions become data-backed:

1. **Setting a price.** If serving a feature costs 0.0016 € per use, the price to the customer
   has to cover it with margin; cost per token is the floor of *pricing*.
2. **Spotting loss-making features.** A feature that consumes many tokens (long prompts, lots of
   output) may cost more than it contributes; cost per request reveals it before you scale.
3. **Comparing build vs buy at product level.** On-prem cost per token against an API tariff,
   for that specific workload, decides whether to serve in house or consume externally.

This is the ultimate reason for the whole FinOps track: it is not infrastructure accounting, it
is the quantitative basis for product and architecture decisions. A measured and attributed cost
per token turns "is this profitable?" from an intuition into a calculation.

---

## Detailed comparison: on-prem, European cloud and API

Extending the comparison with scenarios, for the same job (1M tokens):

| Option | Cost/1M tok | Sovereignty | Where it fits |
|---|---|---|---|
| Depreciated on-prem, high utilisation | **~0.7–1.1 €** | full (EU) | high, sustained volume |
| Depreciated on-prem, low utilisation | **~2–4 €** | full (EU) | bad: idle drives cost up |
| European cloud (Scaleway, on-demand) | **~2.2 €** | EU | variable volume, no iron to operate |
| European cloud (reserved/committed) | **~1.5–1.8 €** | EU | medium committed volume |
| External proprietary API | provider tariff | **non-EU** | prototype / no sovereignty requirement |

The table shows two things. First, **on-prem only wins if utilisation is high**, since at low
occupancy idle makes it dearer than cloud, which connects cost per token with scheduling and
utilisation (the idle of A2). Second, the **external API**, however competitive its tariff,
**loses data sovereignty**, an axis that for GDPR data is not negotiable. Cost per token is the
metric that puts the first three on the same scale; sovereignty is the constraint that rules out
the last one for certain kinds of data.

---

## Prompt caching and effective cost

One nuance refines the real cost per token: **prefix caching**. When many requests share the
same prefix (a long system prompt, a repeated context), the engine caches its prefill compute
and **does not recompute it** on every request. That means cached input tokens **cost almost
zero** the second time and thereafter, so the effective cost per token of a workload with a high
cache *hit rate* is **lower** than the nominal cost. External APIs already reflect this with a
reduced price for cached tokens; in self-hosted the effect is that throughput rises (less
prefill to recompute), which by the identity lowers cost per token.

The implication for the cost model: if your workload has shared prefixes (RAG over the same
corpus, agents with the same system prompt), the real cost per token is **lower** than the naive
calculation gives, and it pays to measure the prefix caching *hit rate* so as not to overestimate
cost. And the reverse: a workload of unique, never-repeated prompts gains nothing from the cache,
and its cost per token is the full nominal figure. Cost per token, once again, is not a constant:
it depends on the model, the precision, the utilisation, the goodput **and** the reuse pattern of
the workload. Measuring it on your real traffic, not on a synthetic benchmark without a cache, is
what gives the number you actually pay.

---

## State of the art 2026

- **The gateway as standard**: LiteLLM (and alternatives) are consolidating as the layer that
  measures tokens, attributes cost and governs budgets in the LLM stack.
- **FOCUS for tokens**: layers on top of the gateway generate FOCUS-compatible logs, integrating
  AI cost with the rest of cloud spend (FinOps introduction).
- **Custom pricing as a bridge**: the OpenCost→gateway connection via custom pricing is the
  standard way to give self-hosted models a real cost.
- **Cost per token as a business KPI**: increasingly, cost per token (and per product) is the
  metric the business demands, not the cost of the pod.

---

## Limits and traps (data-driven)

1. **Forgotten custom pricing.** Without declaring to LiteLLM the real cost of your self-hosted
   token, the `response_cost` of your own models is zero or someone else's price. Close the
   OpenCost→gateway bridge.
2. **Static throughput.** Cost per token changes with throughput; if you optimise the engine but
   do not update the custom pricing, the reported cost drifts out of date.
3. **Forgetting prefill/decode.** An average cost per token is fine for reporting; to optimise,
   separate input and output.
4. **Measuring without governing.** Spend logs without budgets or caps make a pretty dashboard;
   the value lies in closing the loop with alerts and caps.
5. **Comparing tariffs with costs without matching assumptions.** Your 1.09 €/1M assumes a
   utilisation and a throughput; the provider's tariff does not. Fix the assumptions before
   comparing.

With cost per token solved, the FinOps track has its central metric; the following articles (A5
multi-tenant chargeback, A8 TCO model) build on it. The full chain, iron (OpenCost) ÷ throughput
(benchmark) = cost per token, measured and attributed by the gateway, is what turns the cost of
AI into a defensible number, in euros and per team.

## Closing

Cost per token is the metric that reconciles the three tracks of the series into a single
number, and its underlying lesson is counter-intuitive: for your own inference, **the cost of a
token is not a price, it is the reflection of your efficiency**. It comes from dividing the cost
of the iron (OpenCost) by useful throughput (the goodput of your benchmark), and it becomes
visible and governable with a gateway (LiteLLM) that counts the tokens, attributes them by team
and applies budgets. The mistake that invalidates the whole exercise is leaving custom pricing at
zero and believing your own inference is "free"; the right move is to close the OpenCost→gateway
bridge and keep the number up to date when the node cost or the throughput changes. Done that
way, cost per token answers with data the one question the business keeps repeating, is it worth
serving this on my own iron, and it does so in euros, per team and per product, comparable
against European cloud and against any API. For a sovereign architecture proposal, it is the
column of the dashboard that translates all the cost engineering into the sentence that closes a
budget meeting: **this is the cost per token, this is how it is measured, and here is the bench
to reproduce it**.

## See also

- [GPU chargeback and showback](https://blog.lo0.es/en/posts/gpu-chargeback-showback-multi-tenancy/index.md) — sharing cost out per team (OpenCost + LiteLLM + Kueue) once you have the cost per token.
- [GPU utilisation as a FinOps lever](https://blog.lo0.es/en/posts/gpu-idle-cost-not-on-any-invoice/index.md) — why cost per token depends on occupancy: the cost of idle.
- [Cloud GPU: a comparison of prices, commitment and sovereign neoclouds](https://blog.lo0.es/en/posts/cloud-gpu-on-demand-reserved-spot-neoclouds-2026-prices/index.md) — the alternative €/GPU-hour: on-demand, spot and reserved prices from European cloud providers against on-premise.
- [TCO of an on-premise GPU cluster: depreciation, energy and infrastructure](https://blog.lo0.es/en/posts/full-tco-on-premise-gpu-cluster-capex-gpu-hour-break-even/index.md) — the reference €/GPU-hour in your own infrastructure: CAPEX, energy, networking and staff with the full spreadsheet.

## Sources

- LiteLLM · Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
- LiteLLM · Custom Pricing (modelos self-hosted) — https://docs.litellm.ai/docs/proxy/custom_pricing
- LiteLLM · Logging — https://docs.litellm.ai/docs/proxy/logging
- LiteLLM · documentación general — https://docs.litellm.ai/docs/
- Statsig · LiteLLM cost tracking — https://www.statsig.com/perspectives/litellm-cost-tracking
- StackPulsar · LiteLLM production monitoring (2026) — https://stackpulsar.com/blog/litellm-production-monitoring/