LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

Tue, 16 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter). Generic example hardware; no real infrastructure.

TL;DR

There are three OSS leaderboards with public data and documented methodology for comparing the energy efficiency of LLMs in inference: Hugging Face AI Energy Score (166 models, Wh/query on H100, 1–5 star scale, launched February 2025), ML.ENERGY Leaderboard v3 (University of Michigan, J/token per task, Zeus tooling, December 2025) and MLPerf Power (samples/joule certified with a physical Yokogawa WT310E power meter). The three measure different dimensions and are not directly interchangeable. The available data shows that reasoning models consume up to 700× more energy than their non-reasoning equivalents; that MoE models consume roughly 3× fewer J/token than a dense model with equivalent active parameters; and that INT4 quantisation cuts consumption by up to 79 % against FP16 under favourable conditions. The inference engine (vLLM vs Transformers) can move the result another 25–40 %. Without fixing hardware, engine, batch size and task, no comparison between leaderboards is valid.

Track context

This article is C5 of the energy pillar. The base context:

The fundamentals of quantisation are a prerequisite for the quantisation section of this article.

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

Field	Detail
URL	huggingface.co/AIEnergyScore · huggingface.co/spaces/AIEnergyScore/Leaderboard
Organisation	Hugging Face (Sasha Luccioni et al.), with Salesforce and Cohere as initial partners
Launch	February 2025 (AI Action Summit, Paris); v2 December 2025
Models indexed	166 (v1, Feb 2025); +39 new ones in v2 (Dec 2025)
Tasks measured	10 tasks: text generation, summarisation, classification, image generation, ASR, audio generation, translation, question answering, reasoning (added in v2)
Unit of measurement	Wh (watt-hour) per 1,000 queries of the task
Reference hardware	NVIDIA H100 exclusively (a single GPU for class A/B models; multiple for class C)
Measurement tooling	CodeCarbon (GPU energy) + the `ai-energy-benchmarks` package (OSS, PyPI)
Rating system	1–5 stars per task: quintiles of the energy range; ⭐⭐⭐⭐⭐ = the 20 % most efficient
Reference batch size	Batch size = 1 (does not reflect production with aggressive batching)
Access for proprietary models	Yes, via an audited Docker container
Update frequency	No fixed cadence; v1 Feb 2025, v2 Dec 2025
Project licence	Apache 2.0 (repository github.com/huggingface/AIEnergyScore)

Scope of the metric. The AI Energy Score measures GPU energy only (CodeCarbon); it does not capture CPU, DRAM or system overhead. The Wh/1k-queries unit covers the whole run time (prefill + decode + framework overhead), but at batch = 1. The results are therefore comparable between models under the same test conditions, but not extrapolable to a production environment with real concurrency without correction.

Model class (the project’s internal classification):

Class	Definition
A	Fits on a consumer GPU (≤ ~24 GB VRAM)
B	Requires a cloud GPU (≥ 40 GB VRAM)
C	Requires multiple GPUs

2 · ML.ENERGY Leaderboard

Field	Detail
URL	ml.energy/leaderboard
Organisation	Symbiotic Lab, University of Michigan (Mosharaf Chowdhury, Jae-Won Chung et al.)
Reference paper	arXiv 2505.06371 — “The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization” (NeurIPS 2025 D&B, Spotlight)
Current version	v3.0 (December 2025)
Measurement tooling	Zeus (github.com/ml-energy/zeus) via NVML + RAPL; measurement overhead in single-digit ms
Unit of measurement	J/token (energy per output token generated) and total energy per complete response
Reference hardware	NVIDIA A100 80 GB and H100 SXM (declared per submission; varies between models)
Tasks measured	6 tasks: chat (text conversation), reasoning, code generation, summarisation, visual question answering, video generation
Normalisation	Mean energy per complete response (prefill + decode). Output J/token is also reported. The task is stated explicitly because it determines the output length
Scope of the measurement	GPU via NVML + CPU/DRAM via RAPL; not a power meter at the wall
Models covered	~40 architectures in the NeurIPS 2025 version; the web leaderboard is updated with more
Licence	Apache 2.0 (zeus: github.com/ml-energy/zeus); MIT (benchmark: github.com/ml-energy/benchmark)
Update frequency	Continuous on the web leaderboard; the paper is a point-in-time snapshot

Zeus as tooling. Zeus is the measurement engine of the ML.ENERGY Leaderboard and also a standalone package (pip install zeus-ml). It supports NVIDIA GPU (NVML), AMD GPU (ROCm), CPU (RAPL), DRAM (RAPL), Apple Silicon and NVIDIA Jetson. The ZeusMonitor adds measurement overhead in single-digit milliseconds. Since May 2025 it has been a PyTorch ecosystem project. MIT licence.

3 · MLPerf Power

The full data sheet is in the C4 article. A summary of the points relevant for comparison with the previous two:

Field	Detail
URL	mlcommons.org/benchmarks/inference-datacenter/
Organisation	MLCommons Power Working Group (>20 orgs)
Unit of measurement	samples/joule (throughput/mean power) = the inverse of J/sample
Hardware	Complete node measured at the wall (AC); Yokogawa WT310E analyser (±0.1 % of reading)
LLM tasks	GPT-J 6B, Llama 2 70B, Mixtral 8×7B (from v5.0)
Granularity	Complete node (GPU + CPU + RAM + fans + PSU losses); no attribution to individual workloads
Node overhead over GPU	25–45 % of total consumption in submissions with a physical analyser
Licence of the corpus	Public results on GitHub (mlcommons/inference_results_vX.Y); PTDaemon requires MLCommons membership

Comparison of the three leaderboards

Dimension	HF AI Energy Score	ML.ENERGY Leaderboard	MLPerf Power
Unit	Wh/1k-queries	Output J/token	samples/J (complete node)
Fixed hardware	H100 (all models)	A100/H100 (varies)	Depends on the submitter
Measurement	CodeCarbon (GPU)	Zeus NVML+RAPL	Physical AC power meter (Yokogawa)
System coverage	GPU only	GPU + CPU + DRAM	Complete node including fans and PSU
Batch size	1	Varies by task	Per LoadGen scenario
Models covered	166+ (text, image, audio)	~40 generative LLMs	Few (GPT-J, Llama 2, Mixtral)
Proprietary models	Yes (audited Docker)	No (OSS only)	Yes (MLCommons members)
External certification	No	No	Yes (SPEC PTDaemon)
Frequency	Point-in-time (v1, v2)	Continuous	Half-yearly (MLPerf rounds)
Licence	Apache 2.0	Apache 2.0 / MIT	Public results; PTDaemon: membership

Incompatibility between leaderboards. The three measure different dimensions: Wh/query ≠ J/token ≠ node samples/J. A direct comparison requires converting units and assuming that the hardware, the engine and the task are equivalent, which is rarely the case across leaderboards.

How energy per token is measured and normalised

The base identity is developed in the C2 article:

$$E_{\text{token}} \,[\text{J/tok}] = \frac{\bar{P} \,[\text{W}]}{\text{throughput} \,[\text{tok/s}]}$$

To compare models against each other, every factor other than the model must be fixed:

Factor	Effect if it varies	How to fix it
Hardware	H100 vs A100 vs L40S changes the result by 2–4×	Declare the exact hardware; compare only within the same HW
Inference engine	vLLM vs Transformers: 25–40 % difference in J/token	Fix the engine and the version
Batch size / concurrency	Batch 1 vs batch 32: throughput rises but so does power; the ratio varies	Declare the batch size; compare within the same regime
Model precision	FP16 vs INT8 vs INT4: up to −79 % energy	Declare the precision; do not mix
Response length	A query with 50 tokens ≠ one with 500	Use a fixed dataset or normalise per token
Measurement window	Including warm-up or idle inflates the numerator	Align the power window with the token window (see C2)

Conversion formula Wh/query ↔ J/token:

$$E_{\text{J/tok}} = \frac{E_{\text{Wh/query}} \times 3600}{\bar{n}_{\text{tokens/query}}}$$

Example: if a model consumes 0.05 Wh/query (= 180 J/query) and generates an average of 200 tokens per query:

$$E_{\text{J/tok}} = \frac{0{.}05 \times 3600}{200} = \frac{180}{200} = 0{.}9 \,\text{J/tok}$$

AI Energy Score data: concrete examples

The v2 data (December 2025, H100 hardware, batch = 1, text generation task with reasoning on/off):

Model	Active params	Reasoning	GPU Wh/1k queries	Stars (text-gen)
DistilGPT-2	82 M	—	1.31	⭐⭐⭐⭐⭐
SmolLM3-3B	3 B	Off	18.35	⭐⭐⭐⭐
SmolLM3-3B	3 B	On	12,791.22	⭐
Phi-4-reasoning-plus	15 B	Off	18.42	⭐⭐⭐⭐
Phi-4-reasoning-plus	15 B	On	9,461.61	⭐
DeepSeek-R1-Distill-Llama-70B	70 B	Off	49.53	⭐⭐⭐
DeepSeek-R1-Distill-Llama-70B	70 B	On	7,626.53	⭐

Source: Hugging Face AI Energy Score v2 (Dec 2025).

The reasoning multiplier. The energy increase when reasoning is switched on ranges from ×154 (DeepSeek-R1-Distill-Llama-70B) to ×697 (SmolLM3-3B). The direct cause: reasoning models generate between 300 and 800 times more tokens than their non-reasoning equivalents (internal chains of thought). The mean across the v2 corpus is ×30 extra energy for reasoning.

Newer models are not always more efficient. Of the 14 comparable models (no reasoning, no MoE, similar size) between the Feb 2025 and Dec 2025 cohorts: 8 out of 14 had equal or higher energy. The range runs from 3 % of the reference model’s energy to almost 2×. Parameter scale is no longer enough to estimate efficiency.

ML.ENERGY Leaderboard data: J/token by family

The data from the arXiv 2505.06371 paper and from leaderboard v3 (A100/H100 hardware, vLLM as the reference engine):

Scale within one family (Llama 3):

Size	Params	Relative J/token (base = 1B)	Params/energy ratio
Llama 3 · 1B	1 B	1.0×	—
Llama 3 · 8B	8 B	~2.1×	8× params → 2.1× energy
Llama 3 · 70B	70 B	~7.3×	70× params → 7.3× energy

The sublinearity (70× params → 7.3× energy, not 70×) reflects that inference energy is dominated by memory bandwidth (memory-bandwidth bound), not by raw FLOPs.

Dense vs MoE:

Model	Type	Total params	Active params/token	Relative J/token
Llama 3 · 8B	Dense	8 B	8 B	1.0×
Mixtral 8×7B	MoE (top-2)	47 B	~13 B	~0.33×
Llama 3 · 70B	Dense	70 B	70 B	~3.5×

The MoE activates only 2 of 8 experts per token. Mixtral 8×7B consumes roughly ⅓ of the J/token of a dense model with 8B active, at a quality comparable to a dense model of larger scale. Routing overhead and holding all the experts in memory cancel out part of the theoretical gain.

Effect of the task (ML.ENERGY v3, same model):

Task	Energy multiplier per response (vs chat)
Chat (text conversation)	1× (reference)
Summarisation	~2–4×
Code generation	~3–6×
Reasoning	~25×
Image + text	1.1–5.2×
Video + text	1.3–15.0×

Reasoning uses ~10× more tokens per response, and the extra memory of the chain of thought reduces the effective batch size, raising energy per token through memory pressure.

Effect of quantisation on energy per token

Data on NVIDIA H100 hardware, Llama 3 family (arXiv 2508.16712 and arXiv 2504.03360):

Precision	Energy reduction vs FP16	Condition
FP16	reference (0 %)	—
BF16	~0 % (iso-energy)	Same hardware and engine
FP8	−25 to −35 %	H100/H200 with native hardware support
INT8	−23 to −44 % (median ~39 %)	Depends on batch size; more at low batches
INT4 (AWQ / GPTQ)	−50 to −79 %	Requires hardware with efficient low-precision support

Warning. On GPUs without native INT4 hardware support (or with suboptimal dequantisation kernels), quantisation can increase latency and energy per token rather than reduce them, because of run-time dequantisation overhead. The benefit of quantisation is real on H100/A100 with TensorRT-LLM or a well-configured llama.cpp, but it is not guaranteed with any engine.

Quantisation and throughput: the memory reduction per model frees VRAM, which allows larger batch sizes. At a larger batch, throughput rises more than power does, cutting J/token further still. The net effect can exceed the direct reduction in energy per operation.

Effect of the inference engine

The engine is a variable that model-level leaderboards tend to fix, but which in production is a decision of your own. Data from published comparisons (vLLM, TensorRT-LLM, naive Transformers, A100):

Engine	Relative J/token vs Transformers baseline
Transformers (naive, not optimised)	1.0× (reference)
vLLM (PagedAttention, continuous batching)	−25 to −35 %
TensorRT-LLM (optimised NVIDIA kernels, FP8)	−35 to −45 %
llama.cpp (hybrid CPU/GPU, INT4)	Variable; −30 to −60 % depending on hardware

Moving from naive Transformers to TensorRT-LLM can cut energy per token by more than moving from a 70B model to an 8B one of the same origin. The choice of engine is a first-order lever on energy efficiency.

Limits of energy leaderboards

Limit	Description
Hardware dependence	A ranking on H100 is not valid on A100 or L40S without correction. The hierarchy of models can change from one piece of hardware to another.
Engine dependence	The results are valid only for the engine they were measured with. A model that is ×2 more efficient on the leaderboard can fall behind if a slower engine is used.
Artificial batch size	AI Energy Score uses batch = 1. In production with aggressive batching, the efficiency relationship between large and small models changes: the large ones scale better with batch.
Training not captured	All the leaderboards measure inference only. The energy cost of training (which can exceed that of inference by 1,000× over the life of the model) is out of scope.
Incompatibility between leaderboards	Wh/query, J/token and samples/J measure different things. Converting between them requires knowing the mean output length, which varies by task and dataset.
Partial system coverage	AI Energy Score and ML.ENERGY measure GPU (+CPU/DRAM with Zeus); they do not capture the overhead of the complete system (PSU losses, fans, interconnect). MLPerf Power does, but covers few models.
Data latency	The leaderboards publish results months after the tests. New hardware (H200, B100, B200) may have no data available at the time of the decision.
No PUE	None of the three includes datacenter PUE. For real TCO, the leaderboard J/token must be multiplied by your own PUE.

Decision table: choosing a model on energy efficiency

The selection criteria in order, without recommendation prose:

Criterion	Question	Action
Task with reasoning	Does the task require step-by-step reasoning?	Yes → multiply the model’s base energy by ×30–700 before comparing. If there is a non-reasoning alternative with sufficient quality, prefer it.
Size vs minimum quality	What is the minimum acceptable quality for the task?	Consult quality benchmarks (see B7 when available). Pick the smallest model that clears the quality threshold; energy grows sublinearly with size.
Dense vs MoE	Does the hardware have enough memory for the full MoE?	If yes: the active-equivalent MoE consumes ~3× fewer J/token than the equivalent dense model. If not: paging or offload eats the gain.
Precision	Does the hardware have native FP8/INT4 support?	H100/H200: native FP8 (−30 %). With TensorRT-LLM: INT4 AWQ (−50 to −79 %). Without native support: stay on FP16 or BF16 until validated with your own benchmark.
Inference engine	Is the optimal engine for the hardware being used?	Measure with C3. If the engine is not optimised, changing engine can cut energy more than changing model.
Consult the leaderboard	Is the task covered by AI Energy Score or ML.ENERGY?	Filter by: same task, same hardware class, reasoning explicitly off/on. Do not compare models from different hardware classes or different engines.
Validate on your own hardware	Are the leaderboard results on the same HW as yours?	Always validate with Zeus or DCGM on your own hardware before taking the final decision. The leaderboard is a reference, not a prediction.

Quick signals table:

Signal	Effect on energy	Data source
Switching reasoning on	×30–700	AI Energy Score v2
Going from dense 8B to dense 70B	~×3.5	ML.ENERGY Leaderboard v3
Going from dense 8B to active-equivalent 8B MoE	~×0.33 (−67 %)	ML.ENERGY v3
FP16 → INT4 (compatible hardware)	−50 to −79 %	arXiv 2508.16712, 2504.03360
Naive Transformers → TensorRT-LLM FP8	−35 to −45 %	TokenPowerBench, ML.ENERGY
PUE 1.0 → PUE 1.5	+50 % on real datacenter energy	MLPerf Power (scope)

Reference data: energy in a generic node (4×H100 SXM)

Generic example hardware, to anchor the leaderboard values to a real node:

Parameter	Indicative value
TDP 4×H100 SXM 80 GB	4 × 700 W = 2,800 W (GPU only)
Complete-node system power (at the wall)	~3,500–5,000 W depending on load
Non-GPU overhead over GPU	25–45 %
J/token Llama 3 70B FP16, vLLM, batch 8	~1–3 J/tok (indicative, A100/H100)
J/token Llama 3 8B FP16, vLLM, batch 8	~0.3–0.7 J/tok (indicative)
J/token Mixtral 8×7B FP16, vLLM, batch 8	~0.4–0.8 J/tok (indicative)
Energy per 1M tokens (Llama 3 70B, PUE 1.4)	~0.5–1.2 kWh

The J/token values are indicative and depend heavily on batch size, prompt length, prefill/decode ratio and engine version. For certified values, consult the MLPerf Power submissions (mlcommons.org).

For the alternative reference node (4×A100 PCIe 80 GB, TDP ~300 W each):

Parameter	Indicative value
TDP 4×A100 PCIe	4 × 300 W = 1,200 W (GPU only)
Complete-node system power	~1,500–2,000 W
J/token Llama 3 70B FP16, vLLM	~2–5 J/tok (indicative; higher because of lower HBM bandwidth vs SXM)

How to use the leaderboards in practice

A decision flow based on the public data available:

Step	Action	Resource
1	Identify the dominant task of the workload	—
2	Consult AI Energy Score filtered by task and hardware class	huggingface.co/spaces/AIEnergyScore/Leaderboard
3	Note the models with ⭐⭐⭐⭐ or ⭐⭐⭐⭐⭐ on the task	Wh/1k-queries as a relative reference
4	Cross-check with ML.ENERGY for the J/token of each candidate	ml.energy/leaderboard
5	If any model is in MLPerf Power (Llama 2, GPT-J, Mixtral), consult the certified samples/J	mlcommons.org/benchmarks/inference-datacenter/
6	Select the 2–3 candidates with the best energy/quality ratio	—
7	Measure on your own hardware with Zeus or DCGM	github.com/ml-energy/zeus
8	Multiply the measured J/token by the datacenter PUE	J/token × PUE = effective J/token in the datacenter
9	Work out the electricity cost per token at the contracted price	See C2

Sources

Hugging Face · AI Energy Score · organización y leaderboard — https://huggingface.co/AIEnergyScore
Hugging Face · Announcing AI Energy Score Ratings (Luccioni et al., feb. 2025) — https://huggingface.co/blog/sasha/announcing-ai-energy-score
Hugging Face · AI Energy Score v2: Refreshed Leaderboard, now with Reasoning (Luccioni, Gamazaychikov, dic. 2025) — https://huggingface.co/blog/sasha/ai-energy-score-v2
Hugging Face · AIEnergyScore GitHub (Apache 2.0) — https://github.com/huggingface/AIEnergyScore
ML.ENERGY Initiative · Leaderboard — https://ml.energy/leaderboard
ML.ENERGY Initiative · Blog: Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (dic. 2025) — https://ml.energy/blog/measurement/energy/diagnosing-inference-energy-consumption-with-the-mlenergy-leaderboard-v30/
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (Chung et al., NeurIPS 2025 D&B Spotlight) — https://arxiv.org/abs/2505.06371
ML.ENERGY Initiative · Zeus: Deep Learning Energy Measurement and Optimization — https://ml.energy/zeus/
GitHub ml-energy/zeus (MIT) — https://github.com/ml-energy/zeus
PyTorch Blog · Zeus: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
University of Michigan CSE · Power-hungry AI: Researchers evaluate energy consumption across models — https://cse.engin.umich.edu/stories/power-hungry-ai-researchers-evaluate-energy-consumption-across-models
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (dic. 2024) — https://arxiv.org/abs/2512.03024
arXiv 2508.16712 · Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective — https://arxiv.org/abs/2508.16712
arXiv 2504.03360 · Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — https://arxiv.org/abs/2504.03360
Epoch AI · AI Energy Use: Data & Research — https://epoch.ai/topics/energy
MLCommons · MLPerf Inference Datacenter benchmark results — https://mlcommons.org/benchmarks/inference-datacenter/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
Coalition for Sustainable AI · AI Energy Score as best practice in benchmarking — https://www.sustainableaicoalition.org/ai-energy-score-a-standardized-approach-to-evaluating-ai-model-energy-efficiency/

Ml-Energy on lo0 — Tech Blog