From GPU-hour to cost per token: the metric that compares on-prem and cloud

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Fourth article in the FinOps track (A4), and the one that closes out the metric that gives the whole pillar its meaning: cost per token. In A2 we saw how OpenCost assigns the cost of the iron (the pod, in €/hour); in A3, which tool to use. But the cost of the pod is not the metric that compares on-prem with cloud, nor the one the business understands: that one is cost per token (or per request). This article explains how it is really calculated (spoiler: for your own inference it is not a list price, it is €/GPU-hour divided by throughput), how to instrument it with a gateway (LiteLLM), how to attribute it by team and product, and how to compare it against European cloud. No recommendations; only the mechanics and the numbers.

The identity: from GPU-hour to token

Cost per million tokens (CPM) is the cost of the iron divided by what it produces:

This is the central formula of the FinOps track, and it has a consequence many people never quite internalise: for a self-hosted model, the cost of a token does not exist as a fixed price. It depends on two things that change: the cost of the iron (from OpenCost, in €/hour) and the throughput (from your benchmark, in tok/s). The same model on the same node costs half as much per token if you double the throughput with an optimisation, because the denominator changes, not the numerator. The cost per token of your own inference is a function of your efficiency, not a tariff.

That sets it radically apart from an external API, where the price per token really is a fixed tariff the provider charges you. Comparing the two requires turning your variable cost into a number, and that is where the iron (A2) meets the tokens (this article).

The two halves of cost per token

Cost per token comes from joining two measurements that live in different places:

The iron half comes from OpenCost (article A2): the by-usage cost of the vLLM pod, in €/hour. The token half comes from the gateway, which intercepts every request and counts tokens per model and per team. Divide, and out comes the cost per token and, crucially, per team, which is what enables chargeback.

The gateway: LiteLLM in depth

The gateway sits between the application and the engine (vLLM) or the provider, and intercepts every request. LiteLLM is the OSS reference: it proxies over 100 models and tracks spend by keys, users and teams, logging spend for all known models automatically (LiteLLM · Spend Tracking).

Its key mechanisms:

With the spend logs table (via the database adapter) you have the detail of every request to build cost dashboards (LiteLLM · Logging), and you can tag each request with team/project/environment for per-team analysis, with notifications (Slack) and hard caps for keys that must not exceed a limit. In practice it is the platform’s token meter and cost allocator.

The self-hosted trap: LiteLLM does not know what your own token costs you

Here is the point almost nobody wires up correctly. LiteLLM tracks cost using list prices for known models (the API ones: OpenAI, Anthropic and so on). But for a model self-hosted on your vLLM there is no list price, because you pay nobody per token. If you tell it nothing, LiteLLM does not know the real cost of your own token. The solution is custom pricing: declaring to LiteLLM the cost per token of your self-hosted model (LiteLLM · Custom Pricing).

And where does that number come from? From the identity above: the cost of the iron (OpenCost) divided by the throughput (your benchmark). That is, the correct flow is:

1. OpenCost gives you the cost of the vLLM pod: ~11 €/hour (A2).
2. Your benchmark gives you the throughput: ~2,800 tok/s (track B).
3. The identity gives the cost per token: 11 ÷ (2,800 × 3,600 / 10⁶) ≈ 1.09 €/1M tok.
4. That 1.09 €/1M goes into LiteLLM as custom pricing.

Only then does the response_cost of each request reflect the real cost of your inference, rather than a zero or somebody else’s list price. Without this step, your LiteLLM cost dashboards lie about your own models. It is the explicit junction of the three tracks: iron (FinOps), throughput (benchmarking) and measurement (gateway).

Configuring custom pricing: an example

In LiteLLM, the real cost of a self-hosted model is declared in its configuration. An example with the cost per token derived from the identity (1.09 €/1M ÷ 10⁶ = 0.00000109 €/token):

# LiteLLM config.yaml — real cost of a self-hosted model
model_list:
  - model_name: llama-3.1-70b-onprem
    litellm_params:
      model: openai/llama-3.1-70b           # your vLLM endpoint
      api_base: http://vllm:8000/v1
      input_cost_per_token:  0.00000109      # €/token (from OpenCost ÷ throughput)
      output_cost_per_token: 0.00000109      # refine if you separate prefill/decode

With that, every response_cost LiteLLM records reflects the real cost of your iron. The operational note: this number must be updated when the cost of the node changes (new depreciation, a different energy price) or the throughput does (an engine optimisation). It is not a constant; it is the output of OpenCost and of the benchmark, and it is recalculated whenever either of the two changes. Automating that recalculation, from node cost and measured throughput to custom pricing, is what keeps cost dashboards honest.

Cost per token changes with throughput: read it off the sweep

Because throughput is the denominator, cost per token is not a point, it is a curve over the concurrency sweep. On the ~11 €/hour node:

Important readings: at low concurrency the cost per token is sky-high (the fixed cost of the node is shared among few tokens); cost falls as occupancy rises. But watch the last row: if you compute cost on raw throughput (3,900) you get 0.78 €, whereas if you compute it on goodput (2,420, the part that meets the SLO) you get 1.26 €, and the latter is the real cost, because requests that violate the SLO do not count as useful production. An honest cost per token is computed on goodput, not on catalogue throughput. This is where the cost track meets the benchmarking one: the correct denominator is goodput.

Cost per request: not only per token

Cost per token is the unit; cost per request is what a complete request bills. A request consumes input tokens (prompt) plus output tokens (generation), and both count:

With nuances: in many engines the prefill (input) and the decode (output) have different costs per token (decode is memory-bound and more expensive per token), so a fine-grained cost model distinguishes the two. For most cases a weighted average cost per token will do; to optimise, it pays to separate prefill and decode. LiteLLM records input and output tokens separately in its spend logs, so the data is there to refine with.

Cost per request is what connects to the product: if a feature consumes 1,500 tokens on average (1,200 of prompt plus 300 of output) and the cost is 1.09 €/1M, each use costs ~0.0016 €, the basis for setting margin, as seen in the FinOps introduction.

Cost per model and per tier

A platform does not serve a single model. Cost per token varies by model and by configuration, and the gateway tells them apart. An 8B and a 70B on the same node have very different costs per token: the 8B yields far more throughput (more tokens/s) for the same GPU-hour, so its cost per token is a fraction of the 70B’s. The design consequence: routing each request to the cheapest model that meets the quality bar is a direct cost lever, since serving a 70B where an 8B suffices means overpaying per token.

The gateway, with its per-model custom pricing, makes this difference visible and enables cost-based routing: classify the request and send it to the appropriate tier. It connects with the L7 router of the serving track: the gateway not only measures cost, it can decide on the basis of it. And quantisation (FP16→FP8) shows up again as a lever: the same model in FP8 raises throughput and lowers cost per token, moving the figure without changing model.

Attribution: from cost to team and product

The reason for putting a gateway in place is not only to measure total cost, it is to share it out. By tagging each request with team, product and environment, LiteLLM builds the split by business dimension:

With per-team budgets and hard caps per key, the gateway does not only report: it governs. A team that goes over budget gets an alert; a key with a hard cap cannot spend any more. It is the FinOps Operate phase applied to the token.

A combined report example

Joining the iron (OpenCost) and the tokens (gateway), the monthly cost-per-token-per-team report on the example node:

What it reveals: A consumes a lot but at an efficient cost (FP8, high utilisation); B uses a cheap model (8B) for its workload; C consumes little but, as we saw in A2, drags along the idle cost of its underused GPU. The token report plus the OpenCost iron report give the full picture: who spends, on which model, with what efficiency. It is the chargeback that neither half would produce alone. And the per-team figure is what gets taken into the budget conversation: not “the cluster costs X”, but “your team consumed Y million tokens at Z €/million”.

Governing spend: budgets, caps and alerts

Measuring without governing is a dashboard; the value lies in closing the loop. LiteLLM allows three levels of control over spend:

The healthy pattern: soft budgets with alerts for the teams (so they adjust their consumption), rate limits to smooth spikes, and hard caps only on critical keys (a service that must not run away). With this, cost per token stops being an after-the-fact figure and becomes a real-time control: the FinOps Operate phase, applied to inference.

On-prem vs European cloud comparison (the reason for all of it)

Cost per token exists to answer one question: is it cheaper to serve on my own iron or to rent? With the identity and European figures:

Cost per token is what makes the incomparable comparable: your variable cost (iron ÷ throughput) against a provider’s fixed tariff. And it exposes the lever: if an optimisation raises your throughput from 2,800 to 4,200 tok/s, your on-prem cost drops to ~0.73 €/1M, widening the advantage over renting. That is why benchmarking (track B) is, indirectly, the tool that moves cost per token the most: every goodput improvement translates into this number.

From cost per token to product price

Cost per token closes the loop with the business when it becomes cost per unit of product. The chain: cost per token (from the identity) × tokens per use of a feature = marginal cost of that feature. With that, three business decisions become data-backed:

1. Setting a price. If serving a feature costs 0.0016 € per use, the price to the customer has to cover it with margin; cost per token is the floor of pricing.
2. Spotting loss-making features. A feature that consumes many tokens (long prompts, lots of output) may cost more than it contributes; cost per request reveals it before you scale.
3. Comparing build vs buy at product level. On-prem cost per token against an API tariff, for that specific workload, decides whether to serve in house or consume externally.

This is the ultimate reason for the whole FinOps track: it is not infrastructure accounting, it is the quantitative basis for product and architecture decisions. A measured and attributed cost per token turns “is this profitable?” from an intuition into a calculation.

Detailed comparison: on-prem, European cloud and API

Extending the comparison with scenarios, for the same job (1M tokens):

The table shows two things. First, on-prem only wins if utilisation is high, since at low occupancy idle makes it dearer than cloud, which connects cost per token with scheduling and utilisation (the idle of A2). Second, the external API, however competitive its tariff, loses data sovereignty, an axis that for GDPR data is not negotiable. Cost per token is the metric that puts the first three on the same scale; sovereignty is the constraint that rules out the last one for certain kinds of data.

Prompt caching and effective cost

One nuance refines the real cost per token: prefix caching. When many requests share the same prefix (a long system prompt, a repeated context), the engine caches its prefill compute and does not recompute it on every request. That means cached input tokens cost almost zero the second time and thereafter, so the effective cost per token of a workload with a high cache hit rate is lower than the nominal cost. External APIs already reflect this with a reduced price for cached tokens; in self-hosted the effect is that throughput rises (less prefill to recompute), which by the identity lowers cost per token.

The implication for the cost model: if your workload has shared prefixes (RAG over the same corpus, agents with the same system prompt), the real cost per token is lower than the naive calculation gives, and it pays to measure the prefix caching hit rate so as not to overestimate cost. And the reverse: a workload of unique, never-repeated prompts gains nothing from the cache, and its cost per token is the full nominal figure. Cost per token, once again, is not a constant: it depends on the model, the precision, the utilisation, the goodput and the reuse pattern of the workload. Measuring it on your real traffic, not on a synthetic benchmark without a cache, is what gives the number you actually pay.

State of the art 2026

• The gateway as standard: LiteLLM (and alternatives) are consolidating as the layer that measures tokens, attributes cost and governs budgets in the LLM stack.
• FOCUS for tokens: layers on top of the gateway generate FOCUS-compatible logs, integrating AI cost with the rest of cloud spend (FinOps introduction).
• Custom pricing as a bridge: the OpenCost→gateway connection via custom pricing is the standard way to give self-hosted models a real cost.
• Cost per token as a business KPI: increasingly, cost per token (and per product) is the metric the business demands, not the cost of the pod.

Limits and traps (data-driven)

1. Forgotten custom pricing. Without declaring to LiteLLM the real cost of your self-hosted token, the response_cost of your own models is zero or someone else’s price. Close the OpenCost→gateway bridge.
2. Static throughput. Cost per token changes with throughput; if you optimise the engine but do not update the custom pricing, the reported cost drifts out of date.
3. Forgetting prefill/decode. An average cost per token is fine for reporting; to optimise, separate input and output.
4. Measuring without governing. Spend logs without budgets or caps make a pretty dashboard; the value lies in closing the loop with alerts and caps.
5. Comparing tariffs with costs without matching assumptions. Your 1.09 €/1M assumes a utilisation and a throughput; the provider’s tariff does not. Fix the assumptions before comparing.

With cost per token solved, the FinOps track has its central metric; the following articles (A5 multi-tenant chargeback, A8 TCO model) build on it. The full chain, iron (OpenCost) ÷ throughput (benchmark) = cost per token, measured and attributed by the gateway, is what turns the cost of AI into a defensible number, in euros and per team.

Closing

Cost per token is the metric that reconciles the three tracks of the series into a single number, and its underlying lesson is counter-intuitive: for your own inference, the cost of a token is not a price, it is the reflection of your efficiency. It comes from dividing the cost of the iron (OpenCost) by useful throughput (the goodput of your benchmark), and it becomes visible and governable with a gateway (LiteLLM) that counts the tokens, attributes them by team and applies budgets. The mistake that invalidates the whole exercise is leaving custom pricing at zero and believing your own inference is “free”; the right move is to close the OpenCost→gateway bridge and keep the number up to date when the node cost or the throughput changes. Done that way, cost per token answers with data the one question the business keeps repeating, is it worth serving this on my own iron, and it does so in euros, per team and per product, comparable against European cloud and against any API. For a sovereign architecture proposal, it is the column of the dashboard that translates all the cost engineering into the sentence that closes a budget meeting: this is the cost per token, this is how it is measured, and here is the bench to reproduce it.

See also

• GPU chargeback and showback — sharing cost out per team (OpenCost + LiteLLM + Kueue) once you have the cost per token.
• GPU utilisation as a FinOps lever — why cost per token depends on occupancy: the cost of idle.
• Cloud GPU: a comparison of prices, commitment and sovereign neoclouds — the alternative €/GPU-hour: on-demand, spot and reserved prices from European cloud providers against on-premise.
• TCO of an on-premise GPU cluster: depreciation, energy and infrastructure — the reference €/GPU-hour in your own infrastructure: CAPEX, energy, networking and staff with the full spreadsheet.

Sources

• LiteLLM · Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
• LiteLLM · Custom Pricing (modelos self-hosted) — https://docs.litellm.ai/docs/proxy/custom_pricing
• LiteLLM · Logging — https://docs.litellm.ai/docs/proxy/logging
• LiteLLM · documentación general — https://docs.litellm.ai/docs/
• Statsig · LiteLLM cost tracking — https://www.statsig.com/perspectives/litellm-cost-tracking
• StackPulsar · LiteLLM production monitoring (2026) — https://stackpulsar.com/blog/litellm-production-monitoring/