Levers for spending fewer watts per token: a quantified catalogue for LLM inference

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

Seven independent levers act on the numerator or the denominator of the identity \(\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}\). Applied in order of cost-benefit: quantisation (FP16→FP8) cuts J/token by 30–50% without changing hardware; power capping to 70% of TDP reduces power by 30% with less than a 10% drop in throughput on memory-bound loads; DVFS in the decode phase saves a further 22–45%; continuous batching lowers J/token by 25–40% against unoptimised stacks; speculative decoding trims up to 29% off J/token at low batch; prefix caching eliminates the prefill recompute for repeated prompts; and scheduling to clean hours shifts the carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100 SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect can reach a 73% reduction against the unoptimised baseline (arXiv 2504.17674).

The reference identity

where \(\overline{P}\) is the average power over the measurement window (see the methodology in energy per token). The levers act on one of the two factors:

• They reduce \(\overline{P}\): power capping, DVFS, scheduling to low-intensity hours.
• They raise throughput (the denominator): quantisation, continuous batching, speculative decoding, prefix caching, chunked prefill, model selection.
• Both at once: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher throughput at similar or lower power).

The expanded formula including infrastructure overhead:

For the lever calculations the work is done on \(\overline{P}_{\text{GPU}}\) for clarity; the PUE factor (see energy per token) amplifies or attenuates the effect of each lever proportionally.

Master table: levers × impact × cost × tool

Note: the percentages are against the effective J/token of the load, not against idle board power. The effects are always measured under conditions of equal output quality (same model, same serving stack, same prompt distribution).

Lever 1: quantisation

Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations), which shortens the time spent reading weights from HBM per generated token. Decode is memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width frees bandwidth for higher throughput or allows a larger batch in VRAM.

Concrete reference figures:

• Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten, baseten.co/blog/33-faster-llm-inference-with-fp8-quantization).
• The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token (arXiv 2504.17674).
• Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined hardware+quantisation reduction: ~90% (MLCommons, arXiv 2504.17674).

Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU, HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task; INT4 GPTQ without fine calibration, up to 2–5%. Validate with LLM energy leaderboards and your own quality benchmarks.

OSS tooling

# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
  --quantization fp8 \
  --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
  --model meta-llama/Llama-3-70B-Instruct \
  --recipe w4a16_awq.yaml \
  --output-dir llama-70b-w4a16/

Lever 2: continuous batching

Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays empty until the end of the batch. Continuous batching inserts new requests as soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the throughput grows. The result is a larger denominator at similar power:

Measured figures (arXiv 2504.17674, TokenPowerBench arXiv 2512.03024):

• vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high batch loads.
• Online continuous batching vs static offline: a further −5% of J/token (small but consistent).
• Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
• Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that, J/token flattens or rises slightly.

OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM: --max-num-seqs (maximum concurrent sequences); tuning it to the available VRAM maximises the effective batch.

Lever 3: power capping

Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the clock (internal DVFS). The efficiency trick: LLM decode is memory-bandwidth-bound; the GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading throughput proportionally.

The power cap vs throughput curve (H100 SXM)

Figures from (arXiv 2604.11391) and NERSC (docs.nersc.gov/jobs/power-capping):

Key finding from arXiv 2604.11391: the H100 reaches its maximum effective memory bandwidth with only ~350 W on memory-bound loads (STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only ~10% extra throughput for each additional 100 W.

J/token formula with a power cap

where \(\eta(P_{\text{cap}})\) is the fraction of power actually consumed under the cap (on memory-bound loads, the GPU no longer consumes the whole cap), and \(f(P_{\text{cap}})\) is the resulting fraction of throughput. In the H100’s 490–560 W zone, \(\eta \approx 0{.}9\) and \(f \approx 0{.}92\text{–}0{.}95\): the numerator falls more than the denominator, so J/token improves.

Commands

# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active

# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
  sudo nvidia-smi -i $i -pl 490
done

With DCGM in production, the cap can be set via dcgmi policy --set powercap:490 or through the NVIDIA GPU Operator (the powerLimit field in the DevicePlugin). Zeus (ml.energy/zeus) can sweep caps automatically and measure the resulting J/token per configuration.

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly (nvidia-smi -lgc to lock the GPU clock) or left for the driver to adjust dynamically according to available power and load. In LLM inference, the asymmetry between phases is the key fact:

• Prefill: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
• Decode: memory-bandwidth-bound. The SM clock barely matters; what rules is HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

The paper arXiv 2501.08219 measures that reducing the GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6% latency increase in the decode phase, because decode is memory-bound and the SM frequency is not the bottleneck.

Commands (manual DVFS)

# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without overrunning the latency budget.

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

J/token scales with model size, but sublinearly. For Llama-3 (arXiv 2504.17674):

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware efficiency (caches, tensor parallelism, kernels) cushions the scaling. Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for the task.

MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which reduces active compute. But there is an implementation trap:

• OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes 54.24% more energy per token than the dense model of the same active size (arXiv 2504.17674), because the fused expert kernels are ~19–63% slower than an equivalent dense GEMM.
• OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is equal total parameters, fewer active (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison is equal active parameters per token.

Model selection table by use case

Lever 6: speculative decoding

Mechanics

Speculative decoding uses a small draft model to propose several tokens, which the target model verifies in parallel in a single forward pass. If the tokens are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

The energy efficiency condition

The J/token benefit depends on batch size (arXiv 2602.09113):

Conclusion: speculative decoding is a J/token lever only in a low-latency regime (batch≤16); in high-throughput serving it is counterproductive in J/token terms (although it reduces TPOT latency).

OSS tooling

vLLM supports speculative decoding with --speculative-model and --num-speculative-tokens. TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as draft for Llama-3 70B).

Lever 7: prefix caching and chunked prefill

Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document context) is identical across requests. Storing and reusing it eliminates the prefill recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt and 200-token user prompts, the saving is ~1,000/(1,000+200) = ~83% of prefill energy. In total J/token, the effect depends on the length of the shared prompt against the generation length.

Enabling it in vLLM:

vllm serve meta-llama/Llama-3-70B-Instruct \
  --enable-prefix-caching \
  --max-num-seqs 256

Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with decode tokens from other requests. The benefit: the prefill of a long prompt no longer blocks the decode of other requests (reducing TPOT spikes), which allows higher GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node’s peak power (fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less J/token on mixed loads with long prompts (arXiv 2308.16369).

# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
  --enable-chunked-prefill \
  --max-num-batched-tokens 512

Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the gCO₂/token of deferrable load. The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting (ACM SLIT, arXiv 2507.09942):

• CO₂ reduction from temporal shifting: up to 34.7% for flexible loads.
• With carbon-aware reinforcement learning (Eco-Orchestrator): up to a 70% offset in integrated deployments.
• In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the same day (see from watt to carbon).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a carbon-deadline annotation (the maximum tolerable wait) and the scheduler places them in the minimum-intensity hours within that window.

# Kueue annotation for carbon-aware scheduling
metadata:
  annotations:
    kueue.x-k8s.io/carbon-aware: "true"
    kueue.x-k8s.io/carbon-deadline: "8h"

The power cap vs throughput curve: the sweet spot

Reading the chart: the power curve (solid line) falls faster than the throughput curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W gives only ~5% extra throughput; below 420 W, throughput starts falling faster than power. The efficiency sweet spot (minimum J/token) sits at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill saturates SM compute and needs more power to sustain throughput.

How to measure the effect and validate quality

Measurement stack

The full stack for measuring the effect of each lever:

See the full stack in energy tooling.

Measurement protocol per lever

# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
  /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)

With Zeus, the configuration sweep is automatic:

from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
    # run the benchmark with vLLM FP8
    run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")

Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness):

# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
  --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
  --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/

The MMLU score difference between FP16 and FP8 with calibration is typically <1%. Without calibration it can be 2–4%.

Pareto decision table (energy / latency / quality)

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement · ~ neutral · ↑ worsens · ↑↑ worsens significantly.

Stacking levers: the combined effect

The levers are mostly orthogonal and they stack. The maximum combined impact documented in (arXiv 2504.17674):

“Correct application of the relevant inference optimisations can reduce total energy consumption by up to 73% against unoptimised baselines.”

Recommended sequence by cost-benefit:

1. Optimised engine (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
2. FP8 quantisation (if the hardware supports it: H100, A100 with limited support): a further −30–50% on the remaining J/token.
3. Power cap at 70% TDP: −20–30% power with <10% of throughput. Low cost.
4. Prefix caching: turn it on if there are system prompts or repeated contexts. No cost.
5. Chunked prefill: if the load has long variable prompts.
6. DVFS in decode: a further −22–45% if the SLO allows it.
7. Speculative decoding: only if the batch is low (<16) and TTFT latency matters.
8. Model selection: move to an SLM if the quality is sufficient (evaluate with lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

The values are illustrative estimates based on the cited papers applied to a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

Reference hardware: TDP and power cap ranges

For the exact ranges: nvidia-smi -q -d POWER | grep -E "Min|Max|Current".

Executive summary by role

See the full measurement stack in energy tooling and the J/token methodology in energy per token. For the context of reference energy leaderboards and benchmarks, see LLM energy leaderboards. The connection with full TCO is in GPU utilisation as FinOps.

Sources

• arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
• arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
• arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
• arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
• arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
• arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
• arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
• HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
• arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
• arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
• arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
• Zeus Project (ml.energy) — https://ml.energy/zeus/
• Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
• Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
• NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
• NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
• MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
• vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
• Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
• EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139