Levers for spending fewer watts per token: a quantified catalogue for LLM inference

Tue, 16 Jun 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

Seven independent levers act on the numerator or the denominator of the identity $\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}$. Applied in order of cost-benefit: quantisation (FP16→FP8) cuts J/token by 30–50% without changing hardware; power capping to 70% of TDP reduces power by 30% with less than a 10% drop in throughput on memory-bound loads; DVFS in the decode phase saves a further 22–45%; continuous batching lowers J/token by 25–40% against unoptimised stacks; speculative decoding trims up to 29% off J/token at low batch; prefix caching eliminates the prefill recompute for repeated prompts; and scheduling to clean hours shifts the carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100 SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect can reach a 73% reduction against the unoptimised baseline (arXiv 2504.17674).

The reference identity

$$\text{J/token} = \frac{\overline{P}\ [\text{W}]}{\text{throughput}\ [\text{tok/s}]}$$

where $\overline{P}$ is the average power over the measurement window (see the methodology in energy per token). The levers act on one of the two factors:

They reduce $\overline{P}$: power capping, DVFS, scheduling to low-intensity hours.
They raise throughput (the denominator): quantisation, continuous batching, speculative decoding, prefix caching, chunked prefill, model selection.
Both at once: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher throughput at similar or lower power).

The expanded formula including infrastructure overhead:

$$\text{J/token}_{\text{effective}} = \frac{\overline{P}_{\text{GPU}} + \overline{P}_{\text{rest of node}}}{\text{throughput}_{\text{tok/s}}} \times \text{PUE}$$

For the lever calculations the work is done on $\overline{P}_{\text{GPU}}$ for clarity; the PUE factor (see energy per token) amplifies or attenuates the effect of each lever proportionally.

Master table: levers × impact × cost × tool

Lever	J/token reduction (%)	Effect on latency	Effect on quality	Main OSS tool	Source
Quantisation FP16→FP8	30–50%	−8–33% TPOT (improvement)	<1% degradation on benchmarks	vLLM, TRT-LLM, llm-compressor	arXiv 2504.17674, baseten.co
Quantisation FP16→INT4 (AWQ/GPTQ)	45–60%	latency improves (lower BW)	1–3% degradation depending on model	AutoAWQ, llm-compressor, vLLM	arXiv 2504.17674
Continuous vs static batching	25–40% vs PyTorch	+5× throughput (improvement)	no impact	vLLM, TGI, TRT-LLM	arXiv 2504.17674, TokenPowerBench
Power capping 70% TDP	20–30% power	<10% throughput (decode)	no impact	nvidia-smi, DCGM, Zeus	arXiv 2604.11391, NERSC docs
DVFS decode-phase	22–45% in decode	1–6% total latency	no impact	nvidia-smi, GreenLLM, DVFS-GPT	arXiv 2501.08219, HAL 05190051
Speculative decoding	up to 29% (batch≤16)	TTFT improves	no demonstrable impact	vLLM, TRT-LLM	arXiv 2602.09113
Prefix caching	50–90% on reusable prefill	TTFT −50–80% (repeated prompts)	no impact	vLLM, SGLang	vLLM docs
Chunked prefill	~10–15% on mixed loads	more stable TTFT	no impact	vLLM (SARATHI)	arXiv 2308.16369
SLM vs dense LLM selection	60–80% (7B vs 70B model)	low latency	degradation varies by task	Ollama, vLLM, llama.cpp	arXiv 2504.17674
Scheduling to clean hours	0% J/token, up to −70% CO₂	no impact on the load	no impact	Kueue, Volcano, cron	ACM SLIT, arXiv 2507.09942

Note: the percentages are against the effective J/token of the load, not against idle board power. The effects are always measured under conditions of equal output quality (same model, same serving stack, same prompt distribution).

Lever 1: quantisation

Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations), which shortens the time spent reading weights from HBM per generated token. Decode is memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width frees bandwidth for higher throughput or allows a larger batch in VRAM.

Format	Bits/weight	BW reduction vs FP32	J/token vs FP16	Tool
FP32	32	1× (reference)	+100% (worse)	—
BF16/FP16	16	2×	reference	vLLM native
FP8 (W8A8)	8	~4×	−30–50%	vLLM, TRT-LLM, llm-compressor
INT4 AWQ (W4A16)	4	~8×	−45–60%	AutoAWQ, llm-compressor
INT4 GPTQ (W4A16)	4	~8×	−40–55%	AutoGPTQ, vLLM

Concrete reference figures:

Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten, baseten.co/blog/33-faster-llm-inference-with-fp8-quantization).
The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token (arXiv 2504.17674).
Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined hardware+quantisation reduction: ~90% (MLCommons, arXiv 2504.17674).

Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU, HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task; INT4 GPTQ without fine calibration, up to 2–5%. Validate with LLM energy leaderboards and your own quality benchmarks.

OSS tooling

# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --quantization fp8 \
 --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
 --model meta-llama/Llama-3-70B-Instruct \
 --recipe w4a16_awq.yaml \
 --output-dir llama-70b-w4a16/

Lever 2: continuous batching

Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays empty until the end of the batch. Continuous batching inserts new requests as soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the throughput grows. The result is a larger denominator at similar power:

$$\text{J/token}_{\text{continuous batch}} = \frac{\overline{P}}{\text{throughput}_{\uparrow}}$$

Measured figures (arXiv 2504.17674, TokenPowerBench arXiv 2512.03024):

vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high batch loads.
Online continuous batching vs static offline: a further −5% of J/token (small but consistent).
Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that, J/token flattens or rises slightly.

OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM: --max-num-seqs (maximum concurrent sequences); tuning it to the available VRAM maximises the effective batch.

Lever 3: power capping

Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the clock (internal DVFS). The efficiency trick: LLM decode is memory-bandwidth-bound; the GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading throughput proportionally.

The power cap vs throughput curve (H100 SXM)

Figures from (arXiv 2604.11391) and NERSC (docs.nersc.gov/jobs/power-capping):

Power cap (W)	% of TDP (700 W)	Relative throughput (memory-bound)	Efficiency (tok/W)
700 W	100%	100%	1.00×
560 W	80%	~95%	~1.19×
490 W	70%	~90–95%	~1.27× (sweet spot)
420 W	60%	~80–85%	~1.33× (latency rises)
350 W	50%	~75% (the H100 reaches maximum BW at 350 W memory-bound)	~1.50×
200 W	29%	~50%	frequency saturation

Key finding from arXiv 2604.11391: the H100 reaches its maximum effective memory bandwidth with only ~350 W on memory-bound loads (STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only ~10% extra throughput for each additional 100 W.

J/token formula with a power cap

$$\text{J/token}(P_{\text{cap}}) = \frac{P_{\text{cap}} \times \eta(P_{\text{cap}})}{\text{throughput}_{\text{base}} \times f(P_{\text{cap}})}$$

where $\eta(P_{\text{cap}})$ is the fraction of power actually consumed under the cap (on memory-bound loads, the GPU no longer consumes the whole cap), and $f(P_{\text{cap}})$ is the resulting fraction of throughput. In the H100’s 490–560 W zone, $\eta \approx 0{.}9$ and $f \approx 0{.}92\text{–}0{.}95$: the numerator falls more than the denominator, so J/token improves.

Commands

# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active

# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
 sudo nvidia-smi -i $i -pl 490
done

With DCGM in production, the cap can be set via dcgmi policy --set powercap:490 or through the NVIDIA GPU Operator (the powerLimit field in the DevicePlugin). Zeus (ml.energy/zeus) can sweep caps automatically and measure the resulting J/token per configuration.

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly (nvidia-smi -lgc to lock the GPU clock) or left for the driver to adjust dynamically according to available power and load. In LLM inference, the asymmetry between phases is the key fact:

Prefill: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
Decode: memory-bandwidth-bound. The SM clock barely matters; what rules is HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

System	Energy saving	Latency penalty	Hardware	Source
DVFS-GPT (HAL)	up to 32% at iso-latency	~1–2%	A100, RTX 4090	HAL 05190051
EcoInfer	up to 25.4% (average 21.5%)	<5%	A100	arXiv MDPI Electronics
DualScale (disaggregated prefill/decode)	22–45% with SLO	1–6%	H100	arXiv 2602.18755
GreenLLM (SLO-aware DVFS)	18–30%	within the SLO	—	arXiv 2508.16449

The paper arXiv 2501.08219 measures that reducing the GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6% latency increase in the decode phase, because decode is memory-bound and the SM frequency is not the bottleneck.

Commands (manual DVFS)

# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without overrunning the latency budget.

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

J/token scales with model size, but sublinearly. For Llama-3 (arXiv 2504.17674):

Model	Parameters	Relative J/token	Factor vs 1B
Llama-3 1B	1B	1.0×	reference
Llama-3 8B	8B	~2.3×	×2.3 (not ×8)
Llama-3 70B	70B	~7.3×	×7.3 (not ×70)
Llama-3 405B	405B	~40×	×40 (not ×405)

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware efficiency (caches, tensor parallelism, kernels) cushions the scaling. Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for the task.

MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which reduces active compute. But there is an implementation trap:

OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes 54.24% more energy per token than the dense model of the same active size (arXiv 2504.17674), because the fused expert kernels are ~19–63% slower than an equivalent dense GEMM.
OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is equal total parameters, fewer active (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison is equal active parameters per token.

Model selection table by use case

Use case	Recommended model	Efficiency rationale
Simple chat, entity extraction	SLM 7–8B FP8	−68% J/token vs 70B
RAG with long contexts	70B FP8 or MoE 8×7B	optimal quality/J at K>2K tokens
Complex code, reasoning	70B FP8 or 405B INT4	quality is non-negotiable
Classification, embeddings	dedicated model (e.g. E5, BGE)	orders of magnitude less J
Deferrable offline batch	large MoE (DeepSeek, Mixtral)	high quality, batch throughput

Lever 6: speculative decoding

Mechanics

Speculative decoding uses a small draft model to propose several tokens, which the target model verifies in parallel in a single forward pass. If the tokens are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

The energy efficiency condition

The J/token benefit depends on batch size (arXiv 2602.09113):

Batch size	Effect on J/token	Reason
1–16 (low batch)	−up to 29%	the draft model fills the waiting time; verification is cheap
32–64 (medium batch)	~0% (neutral)	verification overhead cancels out the gain
128+ (high batch)	+25% (worse)	GPU saturated; the draft model adds overhead without saving

Conclusion: speculative decoding is a J/token lever only in a low-latency regime (batch≤16); in high-throughput serving it is counterproductive in J/token terms (although it reduces TPOT latency).

OSS tooling

vLLM supports speculative decoding with --speculative-model and --num-speculative-tokens. TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as draft for Llama-3 70B).

Lever 7: prefix caching and chunked prefill

Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document context) is identical across requests. Storing and reusing it eliminates the prefill recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt and 200-token user prompts, the saving is ~1,000/(1,000+200) = ~83% of prefill energy. In total J/token, the effect depends on the length of the shared prompt against the generation length.

Enabling it in vLLM:

vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-prefix-caching \
 --max-num-seqs 256

Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with decode tokens from other requests. The benefit: the prefill of a long prompt no longer blocks the decode of other requests (reducing TPOT spikes), which allows higher GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node’s peak power (fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less J/token on mixed loads with long prompts (arXiv 2308.16369).

# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-chunked-prefill \
 --max-num-batched-tokens 512

Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the gCO₂/token of deferrable load. The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting (ACM SLIT, arXiv 2507.09942):

CO₂ reduction from temporal shifting: up to 34.7% for flexible loads.
With carbon-aware reinforcement learning (Eco-Orchestrator): up to a 70% offset in integrated deployments.
In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the same day (see from watt to carbon).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a carbon-deadline annotation (the maximum tolerable wait) and the scheduler places them in the minimum-intensity hours within that window.

# Kueue annotation for carbon-aware scheduling
metadata:
 annotations:
 kueue.x-k8s.io/carbon-aware: "true"
 kueue.x-k8s.io/carbon-deadline: "8h"

The power cap vs throughput curve: the sweet spot

Reading the chart: the power curve (solid line) falls faster than the throughput curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W gives only ~5% extra throughput; below 420 W, throughput starts falling faster than power. The efficiency sweet spot (minimum J/token) sits at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill saturates SM compute and needs more power to sustain throughput.

How to measure the effect and validate quality

Measurement stack

The full stack for measuring the effect of each lever:

What to measure	Tool	Metric
Real-time GPU power	DCGM (`DCGM_FI_DEV_POWER_USAGE`)	W per GPU, 100ms
J/token in production	Kepler + vLLM tokens	`rate(kepler_joules) / rate(vllm_tokens)`
J/token on the bench	Zeus (`ZeusMonitor`)	J measured during the benchmark
Throughput	vLLM metrics (`vllm:generation_tokens_total`)	tok/s
TPOT latency	vLLM (`vllm:e2e_request_latency`)	ms/tok
Output quality	lm-evaluation-harness	MMLU, HumanEval, etc.

See the full stack in energy tooling.

Measurement protocol per lever

# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
 /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)

With Zeus, the configuration sweep is automatic:

from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
 # run the benchmark with vLLM FP8
 run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")

Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness):

# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/

The MMLU score difference between FP16 and FP8 with calibration is typically <1%. Without calibration it can be 2–4%.

Pareto decision table (energy / latency / quality)

Lever	J/token	TPOT (latency)	Quality	Implementation cost	Reversibility
FP8 quantisation	↓↓↓	↓ (improves)	~ (−<1%)	Low (1 flag)	High
INT4 AWQ quantisation	↓↓↓↓	↓ (improves)	↓ (−1–3%)	Medium (recalibrate)	High
Continuous batching	↓↓↓	~ (neutral on TPOT)	~	Low (new engine)	High
Power cap 70% TDP	↓↓	↑ <10%	~	Very low (1 command)	Immediate
DVFS decode	↓↓↓	↑ <6%	~	Medium (script + tuning)	High
Speculative decoding	↓ (low batch)	↓ (improves TTFT)	~	Medium (draft model)	High
Prefix caching	↓↓ (prefill)	↓↓ (TTFT improves)	~	Low (1 flag)	High
Chunked prefill	↓	↑ stable TPOT	~	Low (1 flag)	High
SLM (7B vs 70B)	↓↓↓↓	↓↓ (improves a lot)	↓↓ (−variable)	High (quality re-eval)	Medium
MoE (vs dense, same active size)	↑ (+54%)	↑	~	—	—
Scheduling to clean hours	~ J/token, ↓↓ CO₂	~ (for deferrable)	~	Medium (scheduler)	High

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement · ~ neutral · ↑ worsens · ↑↑ worsens significantly.

Stacking levers: the combined effect

The levers are mostly orthogonal and they stack. The maximum combined impact documented in (arXiv 2504.17674):

“Correct application of the relevant inference optimisations can reduce total energy consumption by up to 73% against unoptimised baselines.”

Recommended sequence by cost-benefit:

Optimised engine (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
FP8 quantisation (if the hardware supports it: H100, A100 with limited support): a further −30–50% on the remaining J/token.
Power cap at 70% TDP: −20–30% power with <10% of throughput. Low cost.
Prefix caching: turn it on if there are system prompts or repeated contexts. No cost.
Chunked prefill: if the load has long variable prompts.
DVFS in decode: a further −22–45% if the SLO allows it.
Speculative decoding: only if the batch is low (<16) and TTFT latency matters.
Model selection: move to an SLM if the quality is sufficient (evaluate with lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

Configuration	Power (W)	Throughput (tok/s)	J/token
PyTorch FP16, static batching	~2,100	~400	5.25
vLLM FP16, continuous batching	~2,200	~1,800	1.22
vLLM FP8, continuous batching	~2,100	~2,400	0.875
+ power cap 490 W/GPU (1,960 W node)	~1,960	~2,160	0.907
+ prefix caching (40% hit rate)	~1,960	~2,600	0.754
+ DVFS decode 1,200 MHz	~1,500	~2,400	0.625

The values are illustrative estimates based on the cited papers applied to a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

Reference hardware: TDP and power cap ranges

GPU	TDP (W)	Supported cap range	Decode sweet spot
H100 SXM 80GB	700	200–700 W	~490–560 W
H100 PCIe 80GB	350	100–350 W	~245–280 W
A100 SXM 80GB	400	100–400 W	~280–320 W
A100 PCIe 80GB	300	100–300 W	~210–240 W
L40S 48GB	350	100–350 W	~245–280 W
RTX 5090 (reference, PCIe)	575	~100–575 W	~400–460 W

For the exact ranges: nvidia-smi -q -d POWER | grep -E "Min|Max|Current".

Executive summary by role

Role	First action	Second action	Key tool
Cluster operator	`nvidia-smi -pl [70% TDP]` on every node	Enable persistent mode	nvidia-smi, DCGM
Serving engineer	Migrate to vLLM FP8 with continuous batching	Enable prefix caching	vLLM, llm-compressor
Platform architect	Select the smallest model that meets the quality SLO	Design a carbon-based scheduling policy	lm-evaluation-harness, Kueue
MLOps team	Set up a power cap sweep with Zeus	Instrument J/token in Prometheus	Zeus, DCGM exporter

See the full measurement stack in energy tooling and the J/token methodology in energy per token. For the context of reference energy leaderboards and benchmarks, see LLM energy leaderboards. The connection with full TCO is in GPU utilisation as FinOps.

Sources

arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
Zeus Project (ml.energy) — https://ml.energy/zeus/
Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139

Dvfs on lo0 — Tech Blog

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

TL;DR

The reference identity

Master table: levers × impact × cost × tool

Lever 1: quantisation

Mechanics

Quality cost

OSS tooling

Lever 2: continuous batching

Mechanics

OSS tooling

Lever 3: power capping

Mechanics

The power cap vs throughput curve (H100 SXM)

J/token formula with a power cap

Commands

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

Commands (manual DVFS)

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

MoE vs dense

Model selection table by use case

Lever 6: speculative decoding

Mechanics

The energy efficiency condition

OSS tooling

Lever 7: prefix caching and chunked prefill

Prefix caching

Chunked prefill (SARATHI)

Lever 8: scheduling to clean hours

The power cap vs throughput curve: the sweet spot

How to measure the effect and validate quality

Measurement stack

Measurement protocol per lever

Quality validation

Pareto decision table (energy / latency / quality)

Stacking levers: the combined effect

Reference hardware: TDP and power cap ranges

Executive summary by role

Sources