LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency
Contents
Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter). Generic example hardware; no real infrastructure.
TL;DR
There are three OSS leaderboards with public data and documented methodology for comparing the energy efficiency of LLMs in inference: Hugging Face AI Energy Score (166 models, Wh/query on H100, 1–5 star scale, launched February 2025), ML.ENERGY Leaderboard v3 (University of Michigan, J/token per task, Zeus tooling, December 2025) and MLPerf Power (samples/joule certified with a physical Yokogawa WT310E power meter). The three measure different dimensions and are not directly interchangeable. The available data shows that reasoning models consume up to 700× more energy than their non-reasoning equivalents; that MoE models consume roughly 3× fewer J/token than a dense model with equivalent active parameters; and that INT4 quantisation cuts consumption by up to 79 % against FP16 under favourable conditions. The inference engine (vLLM vs Transformers) can move the result another 25–40 %. Without fixing hardware, engine, batch size and task, no comparison between leaderboards is valid.
Track context
This article is C5 of the energy pillar. The base context:
- C1 — State of the art: energy benchmarking of LLM frameworks
- C2 — Energy per token: methodology
- C3 — Measurement tooling in deployment
- C4 — MLPerf Power
The fundamentals of quantisation are a prerequisite for the quantisation section of this article.
The three leaderboards: technical data sheet
1 · Hugging Face AI Energy Score
| Field | Detail |
|---|---|
| URL | huggingface.co/AIEnergyScore · huggingface.co/spaces/AIEnergyScore/Leaderboard |
| Organisation | Hugging Face (Sasha Luccioni et al.), with Salesforce and Cohere as initial partners |
| Launch | February 2025 (AI Action Summit, Paris); v2 December 2025 |
| Models indexed | 166 (v1, Feb 2025); +39 new ones in v2 (Dec 2025) |
| Tasks measured | 10 tasks: text generation, summarisation, classification, image generation, ASR, audio generation, translation, question answering, reasoning (added in v2) |
| Unit of measurement | Wh (watt-hour) per 1,000 queries of the task |
| Reference hardware | NVIDIA H100 exclusively (a single GPU for class A/B models; multiple for class C) |
| Measurement tooling | CodeCarbon (GPU energy) + the ai-energy-benchmarks package (OSS, PyPI) |
| Rating system | 1–5 stars per task: quintiles of the energy range; ⭐⭐⭐⭐⭐ = the 20 % most efficient |
| Reference batch size | Batch size = 1 (does not reflect production with aggressive batching) |
| Access for proprietary models | Yes, via an audited Docker container |
| Update frequency | No fixed cadence; v1 Feb 2025, v2 Dec 2025 |
| Project licence | Apache 2.0 (repository github.com/huggingface/AIEnergyScore) |
Scope of the metric. The AI Energy Score measures GPU energy only (CodeCarbon); it does not capture CPU, DRAM or system overhead. The Wh/1k-queries unit covers the whole run time (prefill + decode + framework overhead), but at batch = 1. The results are therefore comparable between models under the same test conditions, but not extrapolable to a production environment with real concurrency without correction.
Model class (the project’s internal classification):
| Class | Definition |
|---|---|
| A | Fits on a consumer GPU (≤ ~24 GB VRAM) |
| B | Requires a cloud GPU (≥ 40 GB VRAM) |
| C | Requires multiple GPUs |
2 · ML.ENERGY Leaderboard
| Field | Detail |
|---|---|
| URL | ml.energy/leaderboard |
| Organisation | Symbiotic Lab, University of Michigan (Mosharaf Chowdhury, Jae-Won Chung et al.) |
| Reference paper | arXiv 2505.06371 — “The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization” (NeurIPS 2025 D&B, Spotlight) |
| Current version | v3.0 (December 2025) |
| Measurement tooling | Zeus (github.com/ml-energy/zeus) via NVML + RAPL; measurement overhead in single-digit ms |
| Unit of measurement | J/token (energy per output token generated) and total energy per complete response |
| Reference hardware | NVIDIA A100 80 GB and H100 SXM (declared per submission; varies between models) |
| Tasks measured | 6 tasks: chat (text conversation), reasoning, code generation, summarisation, visual question answering, video generation |
| Normalisation | Mean energy per complete response (prefill + decode). Output J/token is also reported. The task is stated explicitly because it determines the output length |
| Scope of the measurement | GPU via NVML + CPU/DRAM via RAPL; not a power meter at the wall |
| Models covered | ~40 architectures in the NeurIPS 2025 version; the web leaderboard is updated with more |
| Licence | Apache 2.0 (zeus: github.com/ml-energy/zeus); MIT (benchmark: github.com/ml-energy/benchmark) |
| Update frequency | Continuous on the web leaderboard; the paper is a point-in-time snapshot |
Zeus as tooling. Zeus is the measurement engine of the ML.ENERGY Leaderboard and also a standalone package (pip install zeus-ml). It supports NVIDIA GPU (NVML), AMD GPU (ROCm), CPU (RAPL), DRAM (RAPL), Apple Silicon and NVIDIA Jetson. The ZeusMonitor adds measurement overhead in single-digit milliseconds. Since May 2025 it has been a PyTorch ecosystem project. MIT licence.
3 · MLPerf Power
The full data sheet is in the C4 article. A summary of the points relevant for comparison with the previous two:
| Field | Detail |
|---|---|
| URL | mlcommons.org/benchmarks/inference-datacenter/ |
| Organisation | MLCommons Power Working Group (>20 orgs) |
| Unit of measurement | samples/joule (throughput/mean power) = the inverse of J/sample |
| Hardware | Complete node measured at the wall (AC); Yokogawa WT310E analyser (±0.1 % of reading) |
| LLM tasks | GPT-J 6B, Llama 2 70B, Mixtral 8×7B (from v5.0) |
| Granularity | Complete node (GPU + CPU + RAM + fans + PSU losses); no attribution to individual workloads |
| Node overhead over GPU | 25–45 % of total consumption in submissions with a physical analyser |
| Licence of the corpus | Public results on GitHub (mlcommons/inference_results_vX.Y); PTDaemon requires MLCommons membership |
Comparison of the three leaderboards
| Dimension | HF AI Energy Score | ML.ENERGY Leaderboard | MLPerf Power |
|---|---|---|---|
| Unit | Wh/1k-queries | Output J/token | samples/J (complete node) |
| Fixed hardware | H100 (all models) | A100/H100 (varies) | Depends on the submitter |
| Measurement | CodeCarbon (GPU) | Zeus NVML+RAPL | Physical AC power meter (Yokogawa) |
| System coverage | GPU only | GPU + CPU + DRAM | Complete node including fans and PSU |
| Batch size | 1 | Varies by task | Per LoadGen scenario |
| Models covered | 166+ (text, image, audio) | ~40 generative LLMs | Few (GPT-J, Llama 2, Mixtral) |
| Proprietary models | Yes (audited Docker) | No (OSS only) | Yes (MLCommons members) |
| External certification | No | No | Yes (SPEC PTDaemon) |
| Frequency | Point-in-time (v1, v2) | Continuous | Half-yearly (MLPerf rounds) |
| Licence | Apache 2.0 | Apache 2.0 / MIT | Public results; PTDaemon: membership |
Incompatibility between leaderboards. The three measure different dimensions: Wh/query ≠ J/token ≠ node samples/J. A direct comparison requires converting units and assuming that the hardware, the engine and the task are equivalent, which is rarely the case across leaderboards.
How energy per token is measured and normalised
The base identity is developed in the C2 article:
$$E_{\text{token}} \,[\text{J/tok}] = \frac{\bar{P} \,[\text{W}]}{\text{throughput} \,[\text{tok/s}]}$$To compare models against each other, every factor other than the model must be fixed:
| Factor | Effect if it varies | How to fix it |
|---|---|---|
| Hardware | H100 vs A100 vs L40S changes the result by 2–4× | Declare the exact hardware; compare only within the same HW |
| Inference engine | vLLM vs Transformers: 25–40 % difference in J/token | Fix the engine and the version |
| Batch size / concurrency | Batch 1 vs batch 32: throughput rises but so does power; the ratio varies | Declare the batch size; compare within the same regime |
| Model precision | FP16 vs INT8 vs INT4: up to −79 % energy | Declare the precision; do not mix |
| Response length | A query with 50 tokens ≠ one with 500 | Use a fixed dataset or normalise per token |
| Measurement window | Including warm-up or idle inflates the numerator | Align the power window with the token window (see C2) |
Conversion formula Wh/query ↔ J/token:
$$E_{\text{J/tok}} = \frac{E_{\text{Wh/query}} \times 3600}{\bar{n}_{\text{tokens/query}}}$$Example: if a model consumes 0.05 Wh/query (= 180 J/query) and generates an average of 200 tokens per query:
$$E_{\text{J/tok}} = \frac{0{.}05 \times 3600}{200} = \frac{180}{200} = 0{.}9 \,\text{J/tok}$$AI Energy Score data: concrete examples
The v2 data (December 2025, H100 hardware, batch = 1, text generation task with reasoning on/off):
| Model | Active params | Reasoning | GPU Wh/1k queries | Stars (text-gen) |
|---|---|---|---|---|
| DistilGPT-2 | 82 M | — | 1.31 | ⭐⭐⭐⭐⭐ |
| SmolLM3-3B | 3 B | Off | 18.35 | ⭐⭐⭐⭐ |
| SmolLM3-3B | 3 B | On | 12,791.22 | ⭐ |
| Phi-4-reasoning-plus | 15 B | Off | 18.42 | ⭐⭐⭐⭐ |
| Phi-4-reasoning-plus | 15 B | On | 9,461.61 | ⭐ |
| DeepSeek-R1-Distill-Llama-70B | 70 B | Off | 49.53 | ⭐⭐⭐ |
| DeepSeek-R1-Distill-Llama-70B | 70 B | On | 7,626.53 | ⭐ |
Source: Hugging Face AI Energy Score v2 (Dec 2025).
The reasoning multiplier. The energy increase when reasoning is switched on ranges from ×154 (DeepSeek-R1-Distill-Llama-70B) to ×697 (SmolLM3-3B). The direct cause: reasoning models generate between 300 and 800 times more tokens than their non-reasoning equivalents (internal chains of thought). The mean across the v2 corpus is ×30 extra energy for reasoning.
Newer models are not always more efficient. Of the 14 comparable models (no reasoning, no MoE, similar size) between the Feb 2025 and Dec 2025 cohorts: 8 out of 14 had equal or higher energy. The range runs from 3 % of the reference model’s energy to almost 2×. Parameter scale is no longer enough to estimate efficiency.
ML.ENERGY Leaderboard data: J/token by family
The data from the arXiv 2505.06371 paper and from leaderboard v3 (A100/H100 hardware, vLLM as the reference engine):
Scale within one family (Llama 3):
| Size | Params | Relative J/token (base = 1B) | Params/energy ratio |
|---|---|---|---|
| Llama 3 · 1B | 1 B | 1.0× | — |
| Llama 3 · 8B | 8 B | ~2.1× | 8× params → 2.1× energy |
| Llama 3 · 70B | 70 B | ~7.3× | 70× params → 7.3× energy |
The sublinearity (70× params → 7.3× energy, not 70×) reflects that inference energy is dominated by memory bandwidth (memory-bandwidth bound), not by raw FLOPs.
Dense vs MoE:
| Model | Type | Total params | Active params/token | Relative J/token |
|---|---|---|---|---|
| Llama 3 · 8B | Dense | 8 B | 8 B | 1.0× |
| Mixtral 8×7B | MoE (top-2) | 47 B | ~13 B | ~0.33× |
| Llama 3 · 70B | Dense | 70 B | 70 B | ~3.5× |
The MoE activates only 2 of 8 experts per token. Mixtral 8×7B consumes roughly ⅓ of the J/token of a dense model with 8B active, at a quality comparable to a dense model of larger scale. Routing overhead and holding all the experts in memory cancel out part of the theoretical gain.
Effect of the task (ML.ENERGY v3, same model):
| Task | Energy multiplier per response (vs chat) |
|---|---|
| Chat (text conversation) | 1× (reference) |
| Summarisation | ~2–4× |
| Code generation | ~3–6× |
| Reasoning | ~25× |
| Image + text | 1.1–5.2× |
| Video + text | 1.3–15.0× |
Reasoning uses ~10× more tokens per response, and the extra memory of the chain of thought reduces the effective batch size, raising energy per token through memory pressure.
Effect of quantisation on energy per token
Data on NVIDIA H100 hardware, Llama 3 family (arXiv 2508.16712 and arXiv 2504.03360):
| Precision | Energy reduction vs FP16 | Condition |
|---|---|---|
| FP16 | reference (0 %) | — |
| BF16 | ~0 % (iso-energy) | Same hardware and engine |
| FP8 | −25 to −35 % | H100/H200 with native hardware support |
| INT8 | −23 to −44 % (median ~39 %) | Depends on batch size; more at low batches |
| INT4 (AWQ / GPTQ) | −50 to −79 % | Requires hardware with efficient low-precision support |
Warning. On GPUs without native INT4 hardware support (or with suboptimal dequantisation kernels), quantisation can increase latency and energy per token rather than reduce them, because of run-time dequantisation overhead. The benefit of quantisation is real on H100/A100 with TensorRT-LLM or a well-configured llama.cpp, but it is not guaranteed with any engine.
Quantisation and throughput: the memory reduction per model frees VRAM, which allows larger batch sizes. At a larger batch, throughput rises more than power does, cutting J/token further still. The net effect can exceed the direct reduction in energy per operation.
Effect of the inference engine
The engine is a variable that model-level leaderboards tend to fix, but which in production is a decision of your own. Data from published comparisons (vLLM, TensorRT-LLM, naive Transformers, A100):
| Engine | Relative J/token vs Transformers baseline |
|---|---|
| Transformers (naive, not optimised) | 1.0× (reference) |
| vLLM (PagedAttention, continuous batching) | −25 to −35 % |
| TensorRT-LLM (optimised NVIDIA kernels, FP8) | −35 to −45 % |
| llama.cpp (hybrid CPU/GPU, INT4) | Variable; −30 to −60 % depending on hardware |
Moving from naive Transformers to TensorRT-LLM can cut energy per token by more than moving from a 70B model to an 8B one of the same origin. The choice of engine is a first-order lever on energy efficiency.
Limits of energy leaderboards
| Limit | Description |
|---|---|
| Hardware dependence | A ranking on H100 is not valid on A100 or L40S without correction. The hierarchy of models can change from one piece of hardware to another. |
| Engine dependence | The results are valid only for the engine they were measured with. A model that is ×2 more efficient on the leaderboard can fall behind if a slower engine is used. |
| Artificial batch size | AI Energy Score uses batch = 1. In production with aggressive batching, the efficiency relationship between large and small models changes: the large ones scale better with batch. |
| Training not captured | All the leaderboards measure inference only. The energy cost of training (which can exceed that of inference by 1,000× over the life of the model) is out of scope. |
| Incompatibility between leaderboards | Wh/query, J/token and samples/J measure different things. Converting between them requires knowing the mean output length, which varies by task and dataset. |
| Partial system coverage | AI Energy Score and ML.ENERGY measure GPU (+CPU/DRAM with Zeus); they do not capture the overhead of the complete system (PSU losses, fans, interconnect). MLPerf Power does, but covers few models. |
| Data latency | The leaderboards publish results months after the tests. New hardware (H200, B100, B200) may have no data available at the time of the decision. |
| No PUE | None of the three includes datacenter PUE. For real TCO, the leaderboard J/token must be multiplied by your own PUE. |
Decision table: choosing a model on energy efficiency
The selection criteria in order, without recommendation prose:
| Criterion | Question | Action |
|---|---|---|
| Task with reasoning | Does the task require step-by-step reasoning? | Yes → multiply the model’s base energy by ×30–700 before comparing. If there is a non-reasoning alternative with sufficient quality, prefer it. |
| Size vs minimum quality | What is the minimum acceptable quality for the task? | Consult quality benchmarks (see B7 when available). Pick the smallest model that clears the quality threshold; energy grows sublinearly with size. |
| Dense vs MoE | Does the hardware have enough memory for the full MoE? | If yes: the active-equivalent MoE consumes ~3× fewer J/token than the equivalent dense model. If not: paging or offload eats the gain. |
| Precision | Does the hardware have native FP8/INT4 support? | H100/H200: native FP8 (−30 %). With TensorRT-LLM: INT4 AWQ (−50 to −79 %). Without native support: stay on FP16 or BF16 until validated with your own benchmark. |
| Inference engine | Is the optimal engine for the hardware being used? | Measure with C3. If the engine is not optimised, changing engine can cut energy more than changing model. |
| Consult the leaderboard | Is the task covered by AI Energy Score or ML.ENERGY? | Filter by: same task, same hardware class, reasoning explicitly off/on. Do not compare models from different hardware classes or different engines. |
| Validate on your own hardware | Are the leaderboard results on the same HW as yours? | Always validate with Zeus or DCGM on your own hardware before taking the final decision. The leaderboard is a reference, not a prediction. |
Quick signals table:
| Signal | Effect on energy | Data source |
|---|---|---|
| Switching reasoning on | ×30–700 | AI Energy Score v2 |
| Going from dense 8B to dense 70B | ~×3.5 | ML.ENERGY Leaderboard v3 |
| Going from dense 8B to active-equivalent 8B MoE | ~×0.33 (−67 %) | ML.ENERGY v3 |
| FP16 → INT4 (compatible hardware) | −50 to −79 % | arXiv 2508.16712, 2504.03360 |
| Naive Transformers → TensorRT-LLM FP8 | −35 to −45 % | TokenPowerBench, ML.ENERGY |
| PUE 1.0 → PUE 1.5 | +50 % on real datacenter energy | MLPerf Power (scope) |
Reference data: energy in a generic node (4×H100 SXM)
Generic example hardware, to anchor the leaderboard values to a real node:
| Parameter | Indicative value |
|---|---|
| TDP 4×H100 SXM 80 GB | 4 × 700 W = 2,800 W (GPU only) |
| Complete-node system power (at the wall) | ~3,500–5,000 W depending on load |
| Non-GPU overhead over GPU | 25–45 % |
| J/token Llama 3 70B FP16, vLLM, batch 8 | ~1–3 J/tok (indicative, A100/H100) |
| J/token Llama 3 8B FP16, vLLM, batch 8 | ~0.3–0.7 J/tok (indicative) |
| J/token Mixtral 8×7B FP16, vLLM, batch 8 | ~0.4–0.8 J/tok (indicative) |
| Energy per 1M tokens (Llama 3 70B, PUE 1.4) | ~0.5–1.2 kWh |
The J/token values are indicative and depend heavily on batch size, prompt length, prefill/decode ratio and engine version. For certified values, consult the MLPerf Power submissions (mlcommons.org).
For the alternative reference node (4×A100 PCIe 80 GB, TDP ~300 W each):
| Parameter | Indicative value |
|---|---|
| TDP 4×A100 PCIe | 4 × 300 W = 1,200 W (GPU only) |
| Complete-node system power | ~1,500–2,000 W |
| J/token Llama 3 70B FP16, vLLM | ~2–5 J/tok (indicative; higher because of lower HBM bandwidth vs SXM) |
How to use the leaderboards in practice
A decision flow based on the public data available:
| Step | Action | Resource |
|---|---|---|
| 1 | Identify the dominant task of the workload | — |
| 2 | Consult AI Energy Score filtered by task and hardware class | huggingface.co/spaces/AIEnergyScore/Leaderboard |
| 3 | Note the models with ⭐⭐⭐⭐ or ⭐⭐⭐⭐⭐ on the task | Wh/1k-queries as a relative reference |
| 4 | Cross-check with ML.ENERGY for the J/token of each candidate | ml.energy/leaderboard |
| 5 | If any model is in MLPerf Power (Llama 2, GPT-J, Mixtral), consult the certified samples/J | mlcommons.org/benchmarks/inference-datacenter/ |
| 6 | Select the 2–3 candidates with the best energy/quality ratio | — |
| 7 | Measure on your own hardware with Zeus or DCGM | github.com/ml-energy/zeus |
| 8 | Multiply the measured J/token by the datacenter PUE | J/token × PUE = effective J/token in the datacenter |
| 9 | Work out the electricity cost per token at the contracted price | See C2 |
See also
- C1 — Energy benchmarking of LLM frameworks: state of the art
- C2 — Energy per token: methodology and the electricity market
- C3 — Measurement tooling in deployment: accuracy and overhead
- C4 — MLPerf Power: the certified standard benchmark
- Fundamentals of quantisation for inference
Sources
- Hugging Face · AI Energy Score · organización y leaderboard — https://huggingface.co/AIEnergyScore
- Hugging Face · Announcing AI Energy Score Ratings (Luccioni et al., feb. 2025) — https://huggingface.co/blog/sasha/announcing-ai-energy-score
- Hugging Face · AI Energy Score v2: Refreshed Leaderboard, now with Reasoning (Luccioni, Gamazaychikov, dic. 2025) — https://huggingface.co/blog/sasha/ai-energy-score-v2
- Hugging Face · AIEnergyScore GitHub (Apache 2.0) — https://github.com/huggingface/AIEnergyScore
- ML.ENERGY Initiative · Leaderboard — https://ml.energy/leaderboard
- ML.ENERGY Initiative · Blog: Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (dic. 2025) — https://ml.energy/blog/measurement/energy/diagnosing-inference-energy-consumption-with-the-mlenergy-leaderboard-v30/
- arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (Chung et al., NeurIPS 2025 D&B Spotlight) — https://arxiv.org/abs/2505.06371
- ML.ENERGY Initiative · Zeus: Deep Learning Energy Measurement and Optimization — https://ml.energy/zeus/
- GitHub ml-energy/zeus (MIT) — https://github.com/ml-energy/zeus
- PyTorch Blog · Zeus: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
- University of Michigan CSE · Power-hungry AI: Researchers evaluate energy consumption across models — https://cse.engin.umich.edu/stories/power-hungry-ai-researchers-evaluate-energy-consumption-across-models
- arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (dic. 2024) — https://arxiv.org/abs/2512.03024
- arXiv 2508.16712 · Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective — https://arxiv.org/abs/2508.16712
- arXiv 2504.03360 · Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — https://arxiv.org/abs/2504.03360
- Epoch AI · AI Energy Use: Data & Research — https://epoch.ai/topics/energy
- MLCommons · MLPerf Inference Datacenter benchmark results — https://mlcommons.org/benchmarks/inference-datacenter/
- arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
- Coalition for Sustainable AI · AI Energy Score as best practice in benchmarking — https://www.sustainableaicoalition.org/ai-energy-score-a-standardized-approach-to-evaluating-ai-model-energy-efficiency/