LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

Notation: amounts in euros (N €), decimals with a full stop. The dollar sign is not used (on this site it is a formula delimiter).

Post B7 of the benchmarking track. The B1 introduction established the distinction between the three axes (performance, quality, energy). The B2 tool catalogue covered the performance axis (TTFT, ITL, goodput). This article covers the quality axis: what each benchmark measures, how it is run, and why the leaderboard numbers do not always say what they appear to say. The LLMOps eval pipeline and LLM-as-judge cover the system’s internal evaluation layer; this article covers the evaluation of the base model with standard academic benchmarks.

TL;DR

An inference engine with an 80 ms TTFT serving a low-quality model is worth nothing: the platform’s Pareto front has two independent axes. In 2025–2026, the most widely used static benchmarks (MMLU, HumanEval) have crossed the saturation threshold (HumanEval > 93% in frontier models, MMLU-Pro > 75% in the top-5), which invalidates their ability to discriminate at the high end. The main cause is not that the models are better: it is data contamination, with 18% n-gram overlap detected in Llama-2-70B against evaluation datasets. The OSS tools active in 2026 are lm-evaluation-harness (EleutherAI, MIT, 60+ tasks, the ecosystem’s de facto engine), HELM (Stanford, Apache 2.0, in maintenance since June 2026), LiveBench (MIT, 1,000 questions refreshed monthly to avoid contamination), lighteval (Hugging Face, MIT, 1,000+ tasks, vLLM support), and LMArena/Chatbot Arena (Bradley-Terry over millions of human comparisons). OpenLLM Leaderboard v2 was archived in 2025 and replaced by the OpenEvals ecosystem (200+ specialised leaderboards). Reproducing an eval number requires pinning: harness version, model version, prompt template version and number of shots.

Quality ≠ performance: the independent Pareto axis

The performance axis (TTFT, ITL, throughput) measures how long the engine takes to produce tokens. The quality axis measures how correct those tokens are. They are orthogonal: a small model served with vLLM at 3,000 tok/s can score 42% on MMLU-Pro; a frontier model may need 4× more GPU and score 72%. The product throughput × quality is the only number that matters for a platform in production.

The operational consequence: select model and engine first by quality (does it clear the minimum threshold for the task?) and then by performance (does it meet the SLO within the GPU budget?). Inverting the order, picking the fastest engine and fitting the model into it, is the most frequent mistake in initial sizing.

Reference benchmarks: what each one measures

The table below collects the standard academic benchmarks in 2026, the capability each one measures, its test set size and the current saturation level in frontier models.

Note on saturation: a saturated benchmark does not distinguish models at the high end; it is only valuable as a floor check (confirming the model has not regressed to the level of three years ago). For the frontier zone, the useful benchmarks in 2026 are MMLU-Pro, GPQA Diamond, IFEval and MuSR.

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

The harness is the industry standard for reproducible evaluation. Its strength is the breadth of tasks and the normalisation of the prompt format. Its limitation: the tasks are static, the datasets do not change between harness versions, which exposes it to the contamination problem if the model was trained on the same data.

Full invocation with vLLM as the backend:

lm_eval \
  --model vllm \
  --model_args pretrained=meta-llama/Llama-3.1-70B-Instruct,dtype=bfloat16 \
  --tasks mmlu_pro,gpqa_diamond,ifeval,humaneval \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path ./eval_results/ \
  --log_samples

HELM (Stanford CRFM)

HELM is the tool that introduced holistic evaluation (multiple metrics over the same scenario instead of a single number). In 2026, with HELM in maintenance mode, the ecosystem has fragmented into specialised leaderboards; but HELM remains the methodological reference for evaluations that need fairness and toxicity as well as accuracy.

HELM Capabilities (published March 2025): an active leaderboard with full prompt transparency, reproducible with the HELM framework. Available at crfm.stanford.edu/helm/capabilities.

LiveBench

LiveBench is the architectural answer to the contamination problem: if the benchmark changes every month with questions based on sources published after the cutoff of the model under evaluation, memorisation of the training set cannot inflate the result. The limitation is size (1,000 questions) and that it does not cover every domain with the same depth as MMLU.

Execution:

git clone https://github.com/LiveBench/LiveBench
cd LiveBench
pip install -e .
python livebench/gen_model_answer.py \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --model-id llama3-70b
python livebench/gen_ground_truth_judgment.py \
  --model-id llama3-70b
python livebench/show_livebench_result.py

lighteval (Hugging Face)

lighteval is the internal successor to lm-evaluation-harness in the HF ecosystem: more modern, better vLLM and distributed backend support, and designed to run without leaving the HF stack. For teams already using the Hub to manage models, it is the lowest-friction option.

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

The OpenLLM Leaderboard v2 evaluated more than 13,000 models between June 2024 and its closure in 2025, using six fixed benchmarks: IFEval, MuSR, GPQA, MATH, BBH and MMLU-Pro, all run with lm-evaluation-harness. Its closure was a direct consequence of the saturation and gaming problem: as the benchmarks became well known, models were optimised for them.

Successor: OpenEvals (huggingface.co/spaces/OpenEvals/find-a-leaderboard)

The archived collection is available for reference at huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025.

LMArena / Chatbot Arena

LMArena is the only benchmark based on human preference at scale. Its advantage: it captures real perceived quality, not just MCQ accuracy. Its limitation: it is hard to reproduce on-premise (it requires a user interface and a panel of human voters). Useful as an external validation signal, not as a CI gate.

Decision table: which tool for which objective

The data contamination trap

What it is

A model that has seen a benchmark’s test data during pretraining memorises answers instead of generalising. The benchmark measures memorisation, not capability. This is called data contamination or benchmark leakage.

Scale of the problem (2024–2025 data)

Detection mechanisms

1. N-gram overlap (the classic method)

Compute the fraction of test set n-grams that appear in the model’s pretraining corpus. A practical threshold: if more than 13% of the test’s 8-grams match the corpus, the result is compromised.

Limit: it only works if you have access to the pretraining corpus. For closed models, it is not applicable.

2. Canary strings (decoy insertion)

Unique random strings are inserted into the evaluation dataset while it is being created. If the model reproduces those strings without having seen them in context, it confirms that it memorised them from pretraining. The technique was proposed in “Extracting Training Data from Large Language Models” (Carlini et al., 2021) and adopted in recent benchmarks.

3. Perturbation and reformulation

Semantically equivalent variants of the test are generated (paraphrasing, changing proper nouns, translation and back-translation). A model that generalises keeps its accuracy; a model that memorises drops. The average drop detected in contaminated models is 9–15 pp on equivalent reformulated prompts.

4. LiveBench (monthly refresh)

The architectural solution: if the questions are generated from sources published after the model’s cutoff (recent arXiv papers, news, new data), memorisation from pretraining cannot help. The model has to reason over information it could never have seen.

5. Kernel divergence analysis (Savelka et al., 2025)

A more sophisticated statistical method: it compares the embedding distribution of the test set with the distribution of the training corpus. An abnormally low divergence indicates that the benchmark and the training corpus come from the same source.

Why static leaderboards saturate

The cycle is predictable:

1. A new benchmark is published (MMLU in 2021, GPQA in 2023).
2. The labs use it as an evaluation target during pretraining and fine-tuning.
3. Performance on that benchmark rises quickly, faster than real capability.
4. The benchmark stops discriminating at the high end.
5. The community needs a new benchmark.

The average time between the publication of a benchmark and its saturation in frontier models has gone from ~36 months (MMLU: 2021–2024) to ~18 months (HumanEval: 2021–2023, GPQA: 2023–2025). Saturation is accelerating.

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

An eval result is only reproducible if it declares:

Few-shot vs zero-shot: the numerical difference

Sensitivity to the number of shots varies dramatically by benchmark:

Publishing an MMLU-Pro result in 0-shot without saying so, when the leaderboard standard is 5-shot, inflates the comparison by ~10 pp. Two results from the same model published with different numbers of shots are not comparable.

Prompt sensitivity

Minor perturbations in the prompt template produce variations of up to ±8 pp on MCQ and ±15 pp on generation tasks. Sources of variance identified in the literature:

• Order of the options in MCQ (position bias): up to ±4 pp.
• Wording of the question (paraphrasing): ±3–8 pp.
• Presence or absence of “Let’s think step by step”: ±5–12 pp on reasoning.
• Language of the system instructions: ±3–10 pp in non-multilingual models.

Why two evals give different results

A common scenario: the paper reports MMLU-Pro 74.3% and the reproduction gives 71.8%. Frequent causes:

The sum of these sources explains gaps of 2–6 pp that are not noise but systematic protocol differences.

The confidence interval formula for comparing two models

To know whether the difference between model A (accuracy \(p_A\)) and model B (\(p_B\)) over a test set of size \(n\) is statistically significant:

Applied to GPQA Diamond (\(n = 198\)), if \(p_A = 0.90\) and \(p_B = 0.88\):

The 95% confidence interval is \([{-0.042},\ {+0.082}]\): it crosses zero. A difference of 2 pp over 198 questions is not statistically distinguishable from noise. To distinguish 90% from 88% with 95% confidence you need:

GPQA Diamond is not that size. Differences of 1–2 pp on GPQA Diamond are statistically invisible.

The quality axis in the scorecard

The platform’s scorecard has three columns: performance (goodput under SLO), quality (score on a reference benchmark), energy (J/token). The quality column is built like this:

A full eval over 4 benchmarks (MMLU-Pro 12,032 + GPQA 198 + IFEval 541 + MBPP 500) with a 70B model on a generic 2×H100 node takes on the order of 4–8 hours depending on batch size and number of shots. The GPU cost is real: at ~5 €/h per amortised card, a full eval costs on the order of 40–80 €. It is not run on every commit; it is run on every model release or major adapter release.

See also

• LLM evals: the layer after tracing — the system’s internal evaluation layer (golden dataset, CI gate, calibrated judge). A necessary complement to the academic benchmarks in this article.
• LLM-as-judge: fundamentals — the LLM judge that evaluates outputs in production; different from an academic benchmark, but sharing the need for calibration and reproducibility.
• Evaluating a RAG: RAGAS and the golden dataset — a specialisation of evaluation for RAG systems; faithfulness and context precision do not appear in the academic benchmarks in this article.
• Benchmark tool catalogue: data sheet by data sheet — the performance axis (TTFT, ITL, goodput); the complementary half of this article in the scorecard.
• LLM benchmarking: state of the art — the B1 introduction with the three axes and the track’s general methodology.

Sources

• EleutherAI · lm-evaluation-harness — https://github.com/EleutherAI/lm-evaluation-harness
• Stanford CRFM · HELM — https://github.com/stanford-crfm/helm
• Stanford CRFM · HELM Capabilities (mar-2025) — https://crfm.stanford.edu/2025/03/20/helm-capabilities.html
• Stanford CRFM · HELM (sitio oficial) — https://crfm.stanford.edu/helm/
• LiveBench · paper arXiv:2406.19314 — https://arxiv.org/abs/2406.19314
• LiveBench · sitio oficial — https://livebench.ai
• LiveBench · GitHub — https://github.com/LiveBench/LiveBench
• NYU Center for Data Science · LiveBench blog — https://nyudatascience.medium.com/livebench-challenging-language-models-with-contamination-free-questions-999b52967ec8
• Hugging Face · lighteval — https://github.com/huggingface/lighteval
• Hugging Face · OpenLLM Leaderboard (archivado) — https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
• Hugging Face · OpenEvals (sucesor) — https://huggingface.co/spaces/OpenEvals/find-a-leaderboard
• Hugging Face · Archived Open LLM Leaderboard 2024-2025 — https://huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025
• LMArena · Chatbot Arena — https://lmarena.ai
• LMSys · blog Chatbot Arena Elo — https://www.lmsys.org/blog/2023-05-03-arena/
• “A Statistical Framework for Ranking LLM-Based Chatbots” · arXiv:2412.18407 — https://arxiv.org/pdf/2412.18407
• “Data Contamination or Genuine Generalization?” (2025) — https://www.suaspress.org/ojs/index.php/AJNS/article/view/v2n2a03
• “When Benchmarks Leak: Inference-Time Decontamination” (2025) · arXiv:2601.19334 — https://arxiv.org/pdf/2601.19334
• “How Contaminated Is Your Benchmark?” (2025) · arXiv:2502.00678 — https://arxiv.org/pdf/2502.00678
• “When AI Benchmarks Plateau” (2026) · arXiv:2602.16763 — https://arxiv.org/pdf/2602.16763
• GPQA · arXiv:2311.12022 — https://arxiv.org/abs/2311.12022
• GPQA Diamond leaderboard · Artificial Analysis — https://artificialanalysis.ai/evaluations/gpqa-diamond
• “The Emperor’s New Clothes in Benchmarking?” (2025) · arXiv:2503.16402 — https://arxiv.org/pdf/2503.16402
• lm-evaluation-harness · CLI reference — https://lm-evaluation-harness.readthedocs.io/running_evals/cli_reference/
• crfm-helm · PyPI — https://pypi.org/project/crfm-helm/