---
title: "Levers for spending fewer watts per token: a quantified catalogue for LLM inference"
url: https://blog.lo0.es/en/posts/levers-fewer-watts-per-token-llm-inference/
markdown: https://blog.lo0.es/en/posts/levers-fewer-watts-per-token-llm-inference/index.md
lang: en
date: 2026-06-16
lastmod: 2026-09-13
author: "José Manuel Román"
description: "A quantified catalogue of energy efficiency levers in on-premise LLM inference: FP8/INT4/AWQ quantisation cuts J/token by 30–50%, power capping to 70% of TDP trims power by 30% with \u003c10% throughput loss, DVFS saves up to 45% in the decode phase. With impact tables, the J/token formula, the power cap vs throughput curve and OSS tooling."
tags: ["energy","efficiency","power-capping","quantization","dvfs","zeus","j-por-token","data-series"]
translations:
  es: https://blog.lo0.es/posts/palancas-eficiencia-energetica-inferencia/index.md
---

# Levers for spending fewer watts per token: a quantified catalogue for LLM inference


> Notation: amounts in **euros (N €)**, decimals with a point. The dollar symbol is not used
> (on this site it is a formula delimiter).

## TL;DR

Seven independent levers act on the numerator or the denominator of the identity
\(\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}\). Applied in order of
cost-benefit: **quantisation** (FP16→FP8) cuts J/token by 30–50% without changing hardware;
**power capping** to 70% of TDP reduces power by 30% with less than a 10% drop in
throughput on memory-bound loads; **DVFS** in the decode phase saves a further 22–45%;
**continuous batching** lowers J/token by 25–40% against unoptimised stacks; **speculative
decoding** trims up to 29% off J/token at low batch; **prefix caching** eliminates the
prefill recompute for repeated prompts; and **scheduling to clean hours** shifts the
carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100
SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an
unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect
can reach a 73% reduction against the unoptimised baseline
([arXiv 2504.17674](https://arxiv.org/pdf/2504.17674)).

---

## The reference identity

$$\text{J/token} = \frac{\overline{P}\ [\text{W}]}{\text{throughput}\ [\text{tok/s}]}$$

where \(\overline{P}\) is the average power over the measurement window (see the methodology in
[energy per token](https://blog.lo0.es/en/posts/energy-per-token-spain-methodology-electricity-market-speculation/index.md)). The levers act on one
of the two factors:

- **They reduce \(\overline{P}\)**: power capping, DVFS, scheduling to low-intensity hours.
- **They raise throughput** (the denominator): quantisation, continuous batching, speculative decoding,
  prefix caching, chunked prefill, model selection.
- **Both at once**: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher
  throughput at similar or lower power).

The expanded formula including infrastructure overhead:

$$\text{J/token}_{\text{effective}} = \frac{\overline{P}_{\text{GPU}} + \overline{P}_{\text{rest of node}}}{\text{throughput}_{\text{tok/s}}} \times \text{PUE}$$

For the lever calculations the work is done on \(\overline{P}_{\text{GPU}}\) for clarity; the
PUE factor (see [energy per token](https://blog.lo0.es/en/posts/energy-per-token-spain-methodology-electricity-market-speculation/index.md)) amplifies or
attenuates the effect of each lever proportionally.

---

## Master table: levers × impact × cost × tool

| Lever | J/token reduction (%) | Effect on latency | Effect on quality | Main OSS tool | Source |
|---|---|---|---|---|---|
| Quantisation FP16→FP8 | **30–50%** | −8–33% TPOT (improvement) | <1% degradation on benchmarks | vLLM, TRT-LLM, llm-compressor | arXiv 2504.17674, baseten.co |
| Quantisation FP16→INT4 (AWQ/GPTQ) | **45–60%** | latency improves (lower BW) | 1–3% degradation depending on model | AutoAWQ, llm-compressor, vLLM | arXiv 2504.17674 |
| Continuous vs static batching | **25–40%** vs PyTorch | +5× throughput (improvement) | no impact | vLLM, TGI, TRT-LLM | arXiv 2504.17674, TokenPowerBench |
| Power capping 70% TDP | **20–30%** power | <10% throughput (decode) | no impact | nvidia-smi, DCGM, Zeus | arXiv 2604.11391, NERSC docs |
| DVFS decode-phase | **22–45%** in decode | 1–6% total latency | no impact | nvidia-smi, GreenLLM, DVFS-GPT | arXiv 2501.08219, HAL 05190051 |
| Speculative decoding | **up to 29%** (batch≤16) | TTFT improves | no demonstrable impact | vLLM, TRT-LLM | arXiv 2602.09113 |
| Prefix caching | **50–90%** on reusable prefill | TTFT −50–80% (repeated prompts) | no impact | vLLM, SGLang | vLLM docs |
| Chunked prefill | ~10–15% on mixed loads | more stable TTFT | no impact | vLLM (SARATHI) | arXiv 2308.16369 |
| SLM vs dense LLM selection | **60–80%** (7B vs 70B model) | low latency | degradation varies by task | Ollama, vLLM, llama.cpp | arXiv 2504.17674 |
| Scheduling to clean hours | 0% J/token, up to −70% CO₂ | no impact on the load | no impact | Kueue, Volcano, cron | ACM SLIT, arXiv 2507.09942 |

> Note: the percentages are against the effective J/token of the load, not against idle board
> power. The effects are always measured under conditions of equal output quality (same model,
> same serving stack, same prompt distribution).

---

## Lever 1: quantisation

### Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations),
which shortens the time spent reading weights from HBM per generated token. Decode is
memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width
frees bandwidth for higher throughput or allows a larger batch in VRAM.

| Format | Bits/weight | BW reduction vs FP32 | J/token vs FP16 | Tool |
|---|---|---|---|---|
| FP32 | 32 | 1× (reference) | +100% (worse) | — |
| BF16/FP16 | 16 | 2× | reference | vLLM native |
| FP8 (W8A8) | 8 | ~4× | **−30–50%** | vLLM, TRT-LLM, llm-compressor |
| INT4 AWQ (W4A16) | 4 | ~8× | **−45–60%** | AutoAWQ, llm-compressor |
| INT4 GPTQ (W4A16) | 4 | ~8× | **−40–55%** | AutoGPTQ, vLLM |

Concrete reference figures:

- Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten,
  [baseten.co/blog/33-faster-llm-inference-with-fp8-quantization](https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/)).
- The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token
  ([arXiv 2504.17674](https://arxiv.org/pdf/2504.17674)).
- Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined
  hardware+quantisation reduction: ~90%
  ([MLCommons, arXiv 2504.17674](https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/)).

### Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU,
HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task;
INT4 GPTQ without fine calibration, up to 2–5%. Validate with [LLM energy leaderboards](https://blog.lo0.es/en/posts/llm-energy-leaderboards-wh-per-token/index.md) and your own quality benchmarks.

### OSS tooling

```bash
# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
  --quantization fp8 \
  --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
  --model meta-llama/Llama-3-70B-Instruct \
  --recipe w4a16_awq.yaml \
  --output-dir llama-70b-w4a16/
```

---

## Lever 2: continuous batching

### Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays
empty until the end of the batch. **Continuous batching** inserts new requests as
soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the
throughput grows. The result is a larger denominator at similar power:

$$\text{J/token}_{\text{continuous batch}} = \frac{\overline{P}}{\text{throughput}_{\uparrow}}$$

Measured figures ([arXiv 2504.17674](https://arxiv.org/pdf/2504.17674),
[TokenPowerBench arXiv 2512.03024](https://arxiv.org/pdf/2512.03024)):

- vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high
  batch loads.
- Online continuous batching vs static offline: a further −5% of J/token (small but
  consistent).
- Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
- Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that,
J/token flattens or rises slightly.

### OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM:
`--max-num-seqs` (maximum concurrent sequences); tuning it to the available VRAM
maximises the effective batch.

---

## Lever 3: power capping

### Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the
clock (internal DVFS). The efficiency trick: LLM decode is **memory-bandwidth-bound**; the
GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading
throughput proportionally.

### The power cap vs throughput curve (H100 SXM)

Figures from ([arXiv 2604.11391](https://arxiv.org/html/2604.11391v1)) and NERSC
([docs.nersc.gov/jobs/power-capping](https://docs.nersc.gov/jobs/power-capping/)):

| Power cap (W) | % of TDP (700 W) | Relative throughput (memory-bound) | Efficiency (tok/W) |
|---|---|---|---|
| 700 W | 100% | 100% | 1.00× |
| 560 W | 80% | ~95% | ~1.19× |
| **490 W** | **70%** | **~90–95%** | **~1.27×** (sweet spot) |
| 420 W | 60% | ~80–85% | ~1.33× (latency rises) |
| 350 W | 50% | ~75% (the H100 reaches maximum BW at 350 W memory-bound) | ~1.50× |
| 200 W | 29% | ~50% | frequency saturation |

Key finding from [arXiv 2604.11391](https://arxiv.org/html/2604.11391v1): **the H100 reaches
its maximum effective memory bandwidth with only ~350 W** on memory-bound loads
(STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces
less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where
performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only
~10% extra throughput for each additional 100 W.

### J/token formula with a power cap

$$\text{J/token}(P_{\text{cap}}) = \frac{P_{\text{cap}} \times \eta(P_{\text{cap}})}{\text{throughput}_{\text{base}} \times f(P_{\text{cap}})}$$

where \(\eta(P_{\text{cap}})\) is the fraction of power actually consumed under the
cap (on memory-bound loads, the GPU no longer consumes the whole cap), and \(f(P_{\text{cap}})\) is
the resulting fraction of throughput. In the H100's 490–560 W zone,
\(\eta \approx 0{.}9\) and \(f \approx 0{.}92\text{–}0{.}95\): the numerator falls more than the
denominator, so J/token improves.

### Commands

```bash
# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active
```

```bash
# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
  sudo nvidia-smi -i $i -pl 490
done
```

With DCGM in production, the cap can be set via
`dcgmi policy --set powercap:490` or through the NVIDIA GPU Operator (the
`powerLimit` field in the DevicePlugin). Zeus ([ml.energy/zeus](https://ml.energy/zeus/))
can sweep caps automatically and measure the resulting J/token per configuration.

---

## Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

### Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly
(`nvidia-smi -lgc` to lock the GPU clock) or left for the driver to adjust dynamically
according to available power and load. In LLM inference, the asymmetry between phases
is the key fact:

- **Prefill**: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
- **Decode**: memory-bandwidth-bound. The SM clock barely matters; what rules is
  HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

| System | Energy saving | Latency penalty | Hardware | Source |
|---|---|---|---|---|
| DVFS-GPT (HAL) | up to **32%** at iso-latency | ~1–2% | A100, RTX 4090 | HAL 05190051 |
| EcoInfer | up to **25.4%** (average 21.5%) | <5% | A100 | arXiv MDPI Electronics |
| DualScale (disaggregated prefill/decode) | **22–45%** with SLO | 1–6% | H100 | arXiv 2602.18755 |
| GreenLLM (SLO-aware DVFS) | **18–30%** | within the SLO | — | arXiv 2508.16449 |

The paper [arXiv 2501.08219](https://arxiv.org/abs/2501.08219) measures that reducing the
GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6%
latency increase in the decode phase, because decode is memory-bound and the SM
frequency is not the bottleneck.

### Commands (manual DVFS)

```bash
# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1
```

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or
DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without
overrunning the latency budget.

---

## Lever 5: model selection (SLM, MoE vs dense)

### The effect of scale

J/token scales with model size, but sublinearly. For Llama-3
([arXiv 2504.17674](https://arxiv.org/pdf/2504.17674)):

| Model | Parameters | Relative J/token | Factor vs 1B |
|---|---|---|---|
| Llama-3 1B | 1B | 1.0× | reference |
| Llama-3 8B | 8B | ~2.3× | ×2.3 (not ×8) |
| Llama-3 70B | 70B | ~7.3× | ×7.3 (not ×70) |
| Llama-3 405B | 405B | ~40× | ×40 (not ×405) |

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware
efficiency (caches, tensor parallelism, kernels) cushions the scaling.
Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for
the task.

### MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which
reduces active compute. But there is an implementation trap:

- OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes **54.24% more**
  energy per token than the dense model of the same active size
  ([arXiv 2504.17674](https://arxiv.org/pdf/2504.17674)), because the fused expert
  kernels are ~19–63% slower than an equivalent dense GEMM.
- OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B
  active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is **equal total parameters, fewer
active** (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison
is **equal active parameters** per token.

### Model selection table by use case

| Use case | Recommended model | Efficiency rationale |
|---|---|---|
| Simple chat, entity extraction | SLM 7–8B FP8 | −68% J/token vs 70B |
| RAG with long contexts | 70B FP8 or MoE 8×7B | optimal quality/J at K>2K tokens |
| Complex code, reasoning | 70B FP8 or 405B INT4 | quality is non-negotiable |
| Classification, embeddings | dedicated model (e.g. E5, BGE) | orders of magnitude less J |
| Deferrable offline batch | large MoE (DeepSeek, Mixtral) | high quality, batch throughput |

---

## Lever 6: speculative decoding

### Mechanics

Speculative decoding uses a small draft model to propose several
tokens, which the target model verifies in parallel in a single forward pass. If the tokens
are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

### The energy efficiency condition

The J/token benefit **depends on batch size** ([arXiv 2602.09113](https://arxiv.org/pdf/2602.09113)):

| Batch size | Effect on J/token | Reason |
|---|---|---|
| 1–16 (low batch) | **−up to 29%** | the draft model fills the waiting time; verification is cheap |
| 32–64 (medium batch) | ~0% (neutral) | verification overhead cancels out the gain |
| 128+ (high batch) | **+25%** (worse) | GPU saturated; the draft model adds overhead without saving |

Conclusion: speculative decoding is a J/token lever only in a **low-latency regime
(batch≤16)**; in high-throughput serving it is counterproductive in J/token terms
(although it reduces TPOT latency).

### OSS tooling

vLLM supports speculative decoding with `--speculative-model` and `--num-speculative-tokens`.
TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as
draft for Llama-3 70B).

---

## Lever 7: prefix caching and chunked prefill

### Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document
context) is identical across requests. Storing and reusing it eliminates the prefill
recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the
repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt
and 200-token user prompts, the saving is ~1,000/(1,000+200) = **~83% of prefill
energy**. In total J/token, the effect depends on the length of the shared prompt against
the generation length.

Enabling it in vLLM:
```bash
vllm serve meta-llama/Llama-3-70B-Instruct \
  --enable-prefix-caching \
  --max-num-seqs 256
```

### Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with
decode tokens from other requests. The benefit: the prefill of a long prompt no longer
blocks the decode of other requests (reducing TPOT spikes), which allows higher
GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node's peak power
(fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less
J/token on mixed loads with long prompts ([arXiv 2308.16369](https://arxiv.org/pdf/2308.16369)).

```bash
# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 512
```

---

## Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the **gCO₂/token** of deferrable load.
The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning
to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting ([ACM SLIT, arXiv 2507.09942](https://arxiv.org/pdf/2507.09942)):

- CO₂ reduction from temporal shifting: **up to 34.7%** for flexible loads.
- With carbon-aware reinforcement learning (Eco-Orchestrator): up to a **70% offset** in
  integrated deployments.
- In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the
  same day (see [from watt to carbon](https://blog.lo0.es/en/posts/watt-to-carbon-pue-grid-intensity-gco2-per-token/index.md)).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity
metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a
`carbon-deadline` annotation (the maximum tolerable wait) and the scheduler places them
in the minimum-intensity hours within that window.

```yaml
# Kueue annotation for carbon-aware scheduling
metadata:
  annotations:
    kueue.x-k8s.io/carbon-aware: "true"
    kueue.x-k8s.io/carbon-deadline: "8h"
```

---

## The power cap vs throughput curve: the sweet spot

> **Diagram:** Power cap vs throughput curve on H100 SXM: the efficiency sweet spot (minimum J/token) sits between 490 and 560 W, where the power reduction outpaces the throughput drop — [see the HTML version](https://blog.lo0.es/en/posts/levers-fewer-watts-per-token-llm-inference/)

Reading the chart: the power curve (solid line) falls faster than the throughput
curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W
gives only ~5% extra throughput; below 420 W, throughput starts
falling faster than power. The **efficiency sweet spot** (minimum J/token) sits
at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill
saturates SM compute and needs more power to sustain throughput.

---

## How to measure the effect and validate quality

### Measurement stack

The full stack for measuring the effect of each lever:

| What to measure | Tool | Metric |
|---|---|---|
| Real-time GPU power | DCGM (`DCGM_FI_DEV_POWER_USAGE`) | W per GPU, 100ms |
| J/token in production | Kepler + vLLM tokens | `rate(kepler_joules) / rate(vllm_tokens)` |
| J/token on the bench | Zeus (`ZeusMonitor`) | J measured during the benchmark |
| Throughput | vLLM metrics (`vllm:generation_tokens_total`) | tok/s |
| TPOT latency | vLLM (`vllm:e2e_request_latency`) | ms/tok |
| Output quality | lm-evaluation-harness | MMLU, HumanEval, etc. |

See the full stack in [energy tooling](https://blog.lo0.es/en/posts/measuring-energy-in-production-kepler-dcgm-stack/index.md).

### Measurement protocol per lever

```promql
# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
  /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)
```

With Zeus, the configuration sweep is automatic:

```python
from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
    # run the benchmark with vLLM FP8
    run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")
```

### Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness
([github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)):

```bash
# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
  --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
  --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/
```

The MMLU score difference between FP16 and FP8 with calibration is typically
<1%. Without calibration it can be 2–4%.

---

## Pareto decision table (energy / latency / quality)

| Lever | J/token | TPOT (latency) | Quality | Implementation cost | Reversibility |
|---|---|---|---|---|---|
| FP8 quantisation | ↓↓↓ | ↓ (improves) | ~ (−<1%) | Low (1 flag) | High |
| INT4 AWQ quantisation | ↓↓↓↓ | ↓ (improves) | ↓ (−1–3%) | Medium (recalibrate) | High |
| Continuous batching | ↓↓↓ | ~ (neutral on TPOT) | ~ | Low (new engine) | High |
| Power cap 70% TDP | ↓↓ | ↑ <10% | ~ | Very low (1 command) | Immediate |
| DVFS decode | ↓↓↓ | ↑ <6% | ~ | Medium (script + tuning) | High |
| Speculative decoding | ↓ (low batch) | ↓ (improves TTFT) | ~ | Medium (draft model) | High |
| Prefix caching | ↓↓ (prefill) | ↓↓ (TTFT improves) | ~ | Low (1 flag) | High |
| Chunked prefill | ↓ | ↑ stable TPOT | ~ | Low (1 flag) | High |
| SLM (7B vs 70B) | ↓↓↓↓ | ↓↓ (improves a lot) | ↓↓ (−variable) | High (quality re-eval) | Medium |
| MoE (vs dense, same active size) | ↑ (+54%) | ↑ | ~ | — | — |
| Scheduling to clean hours | ~ J/token, ↓↓ CO₂ | ~ (for deferrable) | ~ | Medium (scheduler) | High |

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement ·
~ neutral · ↑ worsens · ↑↑ worsens significantly.

---

## Stacking levers: the combined effect

The levers are mostly **orthogonal** and they stack. The maximum combined impact
documented in ([arXiv 2504.17674](https://arxiv.org/pdf/2504.17674)):

> "Correct application of the relevant inference optimisations can reduce
> total energy consumption by up to 73% against unoptimised baselines."

Recommended sequence by cost-benefit:

1. **Optimised engine** (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
2. **FP8 quantisation** (if the hardware supports it: H100, A100 with limited support): a further −30–50%
   on the remaining J/token.
3. **Power cap at 70% TDP**: −20–30% power with <10% of throughput. Low cost.
4. **Prefix caching**: turn it on if there are system prompts or repeated contexts. No cost.
5. **Chunked prefill**: if the load has long variable prompts.
6. **DVFS in decode**: a further −22–45% if the SLO allows it.
7. **Speculative decoding**: only if the batch is low (<16) and TTFT latency matters.
8. **Model selection**: move to an SLM if the quality is sufficient (evaluate with
   lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

| Configuration | Power (W) | Throughput (tok/s) | J/token |
|---|---|---|---|
| PyTorch FP16, static batching | ~2,100 | ~400 | **5.25** |
| vLLM FP16, continuous batching | ~2,200 | ~1,800 | **1.22** |
| vLLM FP8, continuous batching | ~2,100 | ~2,400 | **0.875** |
| + power cap 490 W/GPU (1,960 W node) | ~1,960 | ~2,160 | **0.907** |
| + prefix caching (40% hit rate) | ~1,960 | ~2,600 | **0.754** |
| + DVFS decode 1,200 MHz | ~1,500 | ~2,400 | **0.625** |

> The values are illustrative estimates based on the cited papers applied to
> a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch
> and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

---

## Reference hardware: TDP and power cap ranges

| GPU | TDP (W) | Supported cap range | Decode sweet spot |
|---|---|---|---|
| H100 SXM 80GB | 700 | 200–700 W | ~490–560 W |
| H100 PCIe 80GB | 350 | 100–350 W | ~245–280 W |
| A100 SXM 80GB | 400 | 100–400 W | ~280–320 W |
| A100 PCIe 80GB | 300 | 100–300 W | ~210–240 W |
| L40S 48GB | 350 | 100–350 W | ~245–280 W |
| RTX 5090 (reference, PCIe) | 575 | ~100–575 W | ~400–460 W |

For the exact ranges: `nvidia-smi -q -d POWER | grep -E "Min|Max|Current"`.

---

## Executive summary by role

| Role | First action | Second action | Key tool |
|---|---|---|---|
| Cluster operator | `nvidia-smi -pl [70% TDP]` on every node | Enable persistent mode | nvidia-smi, DCGM |
| Serving engineer | Migrate to vLLM FP8 with continuous batching | Enable prefix caching | vLLM, llm-compressor |
| Platform architect | Select the smallest model that meets the quality SLO | Design a carbon-based scheduling policy | lm-evaluation-harness, Kueue |
| MLOps team | Set up a power cap sweep with Zeus | Instrument J/token in Prometheus | Zeus, DCGM exporter |

See the full measurement stack in [energy tooling](https://blog.lo0.es/en/posts/measuring-energy-in-production-kepler-dcgm-stack/index.md)
and the J/token methodology in [energy per token](https://blog.lo0.es/en/posts/energy-per-token-spain-methodology-electricity-market-speculation/index.md).
For the context of reference energy leaderboards and benchmarks, see [LLM energy leaderboards](https://blog.lo0.es/en/posts/llm-energy-leaderboards-wh-per-token/index.md).
The connection with full TCO is in [GPU utilisation as FinOps](https://blog.lo0.es/en/posts/gpu-idle-cost-not-on-any-invoice/index.md).

---

## Sources

- arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
- arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
- arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
- arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
- arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
- arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
- arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
- HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
- arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
- arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
- arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
- Zeus Project (ml.energy) — https://ml.energy/zeus/
- Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
- Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
- NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
- NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
- MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
- vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
- Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
- EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139