Evaluation on lo0 — Tech Blog

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

Tue, 16 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a full stop. The dollar sign is not used (on this site it is a formula delimiter).

Post B7 of the benchmarking track. The B1 introduction established the distinction between the three axes (performance, quality, energy). The B2 tool catalogue covered the performance axis (TTFT, ITL, goodput). This article covers the quality axis: what each benchmark measures, how it is run, and why the leaderboard numbers do not always say what they appear to say. The LLMOps eval pipeline and LLM-as-judge cover the system’s internal evaluation layer; this article covers the evaluation of the base model with standard academic benchmarks.

TL;DR

An inference engine with an 80 ms TTFT serving a low-quality model is worth nothing: the platform’s Pareto front has two independent axes. In 2025–2026, the most widely used static benchmarks (MMLU, HumanEval) have crossed the saturation threshold (HumanEval > 93% in frontier models, MMLU-Pro > 75% in the top-5), which invalidates their ability to discriminate at the high end. The main cause is not that the models are better: it is data contamination, with 18% n-gram overlap detected in Llama-2-70B against evaluation datasets. The OSS tools active in 2026 are lm-evaluation-harness (EleutherAI, MIT, 60+ tasks, the ecosystem’s de facto engine), HELM (Stanford, Apache 2.0, in maintenance since June 2026), LiveBench (MIT, 1,000 questions refreshed monthly to avoid contamination), lighteval (Hugging Face, MIT, 1,000+ tasks, vLLM support), and LMArena/Chatbot Arena (Bradley-Terry over millions of human comparisons). OpenLLM Leaderboard v2 was archived in 2025 and replaced by the OpenEvals ecosystem (200+ specialised leaderboards). Reproducing an eval number requires pinning: harness version, model version, prompt template version and number of shots.

Quality ≠ performance: the independent Pareto axis

The performance axis (TTFT, ITL, throughput) measures how long the engine takes to produce tokens. The quality axis measures how correct those tokens are. They are orthogonal: a small model served with vLLM at 3,000 tok/s can score 42% on MMLU-Pro; a frontier model may need 4× more GPU and score 72%. The product throughput × quality is the only number that matters for a platform in production.

The operational consequence: select model and engine first by quality (does it clear the minimum threshold for the task?) and then by performance (does it meet the SLO within the GPU budget?). Inverting the order, picking the fastest engine and fitting the model into it, is the most frequent mistake in initial sizing.

Reference benchmarks: what each one measures

The table below collects the standard academic benchmarks in 2026, the capability each one measures, its test set size and the current saturation level in frontier models.

Benchmark	Capability measured	Format	Test set	Frontier saturation 2026
MMLU	Multidisciplinary knowledge (57 subjects)	4-choice MCQ	14,079	High: top-5 > 88%; insufficient separation
MMLU-Pro	MMLU with 10 options, multistep reasoning	10-choice MCQ	12,032	Medium: top-5 between 72–77%; still separates
GSM8K	Primary-school maths reasoning (word problems)	answer generation	1,319	Very high: top models > 97%
MATH	Competition maths (algebra, geometry, calculus, etc.)	LaTeX generation	5,000	Medium-high: top models 85–92%
GPQA Diamond	PhD-level scientific reasoning (biology, physics, chemistry)	4-choice MCQ	198	Emerging: top > 90% in 2026; still separates in 60–90%
BBH (BIG-Bench Hard)	23 tasks where earlier models fell below human level	generation	6,511	Medium: useful for complex reasoning
IFEval	Verifiable instruction following (format, length, language)	generation + rules	541	Low: notable separation between models
HumanEval	Python code generation (164 problems with signature)	generation + execution	164	Very high: top > 93%; saturated
MBPP	Code generation (500 problems without a function signature)	generation + execution	500	High: top models 85–92%
MuSR	Multistep reasoning over long texts	generation	~1,000	Low: useful for extended reasoning
TruthfulQA	Truthfulness: resistance to asserting popular myths	generation + MCQ	817	Medium: visible separation between families

Note on saturation: a saturated benchmark does not distinguish models at the high end; it is only valuable as a floor check (confirming the model has not regressed to the level of three years ago). For the frontier zone, the useful benchmarks in 2026 are MMLU-Pro, GPQA Diamond, IFEval and MuSR.

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

Field	Data
Repository	github.com/EleutherAI/lm-evaluation-harness
Licence	MIT
Maintainer	EleutherAI
Active version (Jun 2026)	0.4.x (`main` branch)
Available tasks	60+ benchmarks, hundreds of subtasks and variants
Supported backends	Hugging Face Transformers, vLLM, litellm, OpenAI-compatible API, GGUF
Task format	YAML + Python; each task defines dataset, template, metric and number of shots
What it measures	MCQ accuracy, exact match, pass@k on code, perplexity, BLEU/ROUGE
How it is invoked	`lm_eval --model hf --model_args pretrained=<model> --tasks mmlu,gsm8k --num_fewshot 5 --output_path results/`
Output	JSON with per-task metrics + configuration metadata
Use in the ecosystem	Official engine of the OpenLLM Leaderboard (v1 and v2); de facto reference in research

The harness is the industry standard for reproducible evaluation. Its strength is the breadth of tasks and the normalisation of the prompt format. Its limitation: the tasks are static, the datasets do not change between harness versions, which exposes it to the contamination problem if the model was trained on the same data.

Full invocation with vLLM as the backend:

lm_eval \
 --model vllm \
 --model_args pretrained=meta-llama/Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu_pro,gpqa_diamond,ifeval,humaneval \
 --num_fewshot 5 \
 --batch_size auto \
 --output_path ./eval_results/ \
 --log_samples

HELM (Stanford CRFM)

Field	Data
Repository	github.com/stanford-crfm/helm
Official site	crfm.stanford.edu/helm
Licence	Apache 2.0
Maintainer	Stanford CRFM
Status (Jun 2026)	Maintenance mode since 1 June 2026
What it measures	Accuracy, calibration, robustness, fairness, toxicity, efficiency (latency/cost)
Task format	Python + YAML; scenarios with multiple simultaneous metrics
Supported backends	API (OpenAI, Anthropic, Cohere, etc.) + local models via Hugging Face
Active variants	HELM Classic, HELM Capabilities, HELM Lite, MedHELM, HELM Safety
Installation	`pip install crfm-helm`
How it is invoked	`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000`
Output	JSON + web UI with a reproducible leaderboard and per-prompt transparency

HELM is the tool that introduced holistic evaluation (multiple metrics over the same scenario instead of a single number). In 2026, with HELM in maintenance mode, the ecosystem has fragmented into specialised leaderboards; but HELM remains the methodological reference for evaluations that need fairness and toxicity as well as accuracy.

HELM Capabilities (published March 2025): an active leaderboard with full prompt transparency, reproducible with the HELM framework. Available at crfm.stanford.edu/helm/capabilities.

LiveBench

Field	Data
Repository	github.com/LiveBench/LiveBench
Official site	livebench.ai
Licence	MIT
Maintainer	NYU, UT Austin, UC Santa Barbara
Paper	arXiv:2406.19314
Size	1,000 active questions (monthly refresh, same total size)
Categories	6: math, coding, reasoning, language, instruction following, data analysis
Tasks	18 subtasks
What it measures	General capabilities over questions with objectively verifiable answers
Anti-contamination	Questions generated from recent sources: arXiv papers, news, IMDb, new datasets
Scoring	No LLM-as-judge: automatically verifiable answer (exact ground truth)
Cadence	New questions monthly; old ones retired to avoid accumulated memorisation

LiveBench is the architectural answer to the contamination problem: if the benchmark changes every month with questions based on sources published after the cutoff of the model under evaluation, memorisation of the training set cannot inflate the result. The limitation is size (1,000 questions) and that it does not cover every domain with the same depth as MMLU.

Execution:

git clone https://github.com/LiveBench/LiveBench
cd LiveBench
pip install -e .
python livebench/gen_model_answer.py \
 --model-path meta-llama/Llama-3.1-70B-Instruct \
 --model-id llama3-70b
python livebench/gen_ground_truth_judgment.py \
 --model-id llama3-70b
python livebench/show_livebench_result.py

lighteval (Hugging Face)

Field	Data
Repository	github.com/huggingface/lighteval
Licence	MIT
Maintainer	Hugging Face (Leaderboard & Evals Team)
Available tasks	1,000+ tasks across multiple domains and languages
Supported backends	Hugging Face Accelerate (CPU/GPU/multi-GPU), Nanotron, vLLM, TGI, OpenAI-compatible endpoints
What it measures	Accuracy, BERT score, BLEU, exact match; same as the harness + HF’s own tasks
Differentiator	Designed to be the engine behind HF’s leaderboards; easy extension with custom tasks
Installation	`pip install lighteval`
How it is invoked	`lighteval accelerate –model_args “pretrained=” –tasks “lighteval
Output	JSON + direct integration with the Hugging Face Hub to publish results

lighteval is the internal successor to lm-evaluation-harness in the HF ecosystem: more modern, better vLLM and distributed backend support, and designed to run without leaving the HF stack. For teams already using the Hub to manage models, it is the lowest-friction option.

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

The OpenLLM Leaderboard v2 evaluated more than 13,000 models between June 2024 and its closure in 2025, using six fixed benchmarks: IFEval, MuSR, GPQA, MATH, BBH and MMLU-Pro, all run with lm-evaluation-harness. Its closure was a direct consequence of the saturation and gaming problem: as the benchmarks became well known, models were optimised for them.

Successor: OpenEvals (huggingface.co/spaces/OpenEvals/find-a-leaderboard)

Aspect	OpenLLM Leaderboard v2	OpenEvals
Architecture	Single centralised leaderboard	200+ specialised community leaderboards
Benchmarks	Fixed (6 tasks)	Variable per leaderboard (math, code, medical, safety, etc.)
Models evaluated	13,000+	Distributed across leaderboards
Eval engine	lm-evaluation-harness	lighteval (most) or harness
Status	Archived with an updated UI (Mar 2025)	Active; you can create your own leaderboard
Advantage	Historical consistency	Specialisation, speed of adaptation

The archived collection is available for reference at huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025.

LMArena / Chatbot Arena

Field	Data
Site	lmarena.ai (rebrand of LMSys Chatbot Arena, January 2026)
Methodology	Anonymous pairwise comparisons: the user votes for the answer they prefer without knowing the model
Statistical model	Bradley-Terry (maximum likelihood over the full history of match-ups; replaced classic Elo for greater stability)
Accumulated votes	> 6 million (Jun 2026)
Variants	Overall, Coding, Math, Hard Prompts, Multimodal — the specialised ones are more reliable
Licence/access	Free (web); vote data available under a CC licence
Limitations	Overall is biased by labs’ selective disclosure and by format gaming; Bradley-Terry was recalibrated in mid-2025 (Style Control), shifting some models by ±20–40 Elo

LMArena is the only benchmark based on human preference at scale. Its advantage: it captures real perceived quality, not just MCQ accuracy. Its limitation: it is hard to reproduce on-premise (it requires a user interface and a panel of human voters). Useful as an external validation signal, not as a CI gate.

Decision table: which tool for which objective

Objective	Tool	Why
Reproduce a standard academic eval	lm-evaluation-harness	De facto engine; maximum compatibility with papers
Holistic evaluation (accuracy + fairness + toxicity)	HELM (+ HELM Capabilities)	The only framework measuring every axis at once
Avoid contamination with recent questions	LiveBench	Monthly refresh; verifiable answers without a judge
Native integration with HF Hub and vLLM	lighteval	vLLM backend, 1,000+ tasks, direct publishing to the Hub
Compare against the public state of the art (OSS)	OpenEvals + lighteval	The successor ecosystem to the OpenLLM Leaderboard
Human preference at scale	LMArena	Bradley-Terry over millions of real comparisons
Custom domain eval (medical, legal, etc.)	lighteval with a custom task	Simpler to extend than the harness
Fast CI gate (<10 min)	lm-eval harness (subset)	Individual tasks with `--tasks ifeval,gpqa` + vLLM batching

The data contamination trap

What it is

A model that has seen a benchmark’s test data during pretraining memorises answers instead of generalising. The benchmark measures memorisation, not capability. This is called data contamination or benchmark leakage.

Scale of the problem (2024–2025 data)

Finding	Source	Figure
N-gram overlap in Llama-2-70B	“Data Contamination or Genuine Generalization?” (2025)	18.1% overlap with eval datasets
Out-of-distribution drop in contaminated large models	Ibid.	−9.4 pp on reformulated prompts vs the original test
HumanEval saturated	Artificial Analysis / multiple 2026 reports	Top models > 93%; separation ≤ 2 pp between the best 5
GPQA Diamond: appearance of indirect contamination	mindstudio.ai analysis 2026	“Data laundering”: questions leaked via online forums

Detection mechanisms

1. N-gram overlap (the classic method)

Compute the fraction of test set n-grams that appear in the model’s pretraining corpus. A practical threshold: if more than 13% of the test’s 8-grams match the corpus, the result is compromised.

\[ \text{contamination ratio} = \frac{|\{g \in \text{test n-grams}\} \cap \{g \in \text{train n-grams}\}|}{|\text{test n-grams}|} \]

Limit: it only works if you have access to the pretraining corpus. For closed models, it is not applicable.

2. Canary strings (decoy insertion)

Unique random strings are inserted into the evaluation dataset while it is being created. If the model reproduces those strings without having seen them in context, it confirms that it memorised them from pretraining. The technique was proposed in “Extracting Training Data from Large Language Models” (Carlini et al., 2021) and adopted in recent benchmarks.

3. Perturbation and reformulation

Semantically equivalent variants of the test are generated (paraphrasing, changing proper nouns, translation and back-translation). A model that generalises keeps its accuracy; a model that memorises drops. The average drop detected in contaminated models is 9–15 pp on equivalent reformulated prompts.

4. LiveBench (monthly refresh)

The architectural solution: if the questions are generated from sources published after the model’s cutoff (recent arXiv papers, news, new data), memorisation from pretraining cannot help. The model has to reason over information it could never have seen.

5. Kernel divergence analysis (Savelka et al., 2025)

A more sophisticated statistical method: it compares the embedding distribution of the test set with the distribution of the training corpus. An abnormally low divergence indicates that the benchmark and the training corpus come from the same source.

Why static leaderboards saturate

The cycle is predictable:

A new benchmark is published (MMLU in 2021, GPQA in 2023).
The labs use it as an evaluation target during pretraining and fine-tuning.
Performance on that benchmark rises quickly, faster than real capability.
The benchmark stops discriminating at the high end.
The community needs a new benchmark.

The average time between the publication of a benchmark and its saturation in frontier models has gone from ~36 months (MMLU: 2021–2024) to ~18 months (HumanEval: 2021–2023, GPQA: 2023–2025). Saturation is accelerating.

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

An eval result is only reproducible if it declares:

Parameter	Example	Why it matters
Harness version	`lm-evaluation-harness==0.4.3`	Changes to the prompt template between versions invalidate the comparison
Exact model version	`meta-llama/Llama-3.1-70B-Instruct@sha256:abc…`	Silent model updates on the Hub change the results
Prompt template	`mmlu_pro_cot_0shot` vs `mmlu_pro_5shot`	The difference between 0-shot and 5-shot can be 5–12 pp
Number of shots	`--num_fewshot 5`	The published standard varies by benchmark; 0-shot and 5-shot are not comparable

Few-shot vs zero-shot: the numerical difference

Sensitivity to the number of shots varies dramatically by benchmark:

Benchmark	Typical 0-shot	Typical 5-shot	Difference
MMLU-Pro	62%	72%	+10 pp
GPQA Diamond	55%	65%	+10 pp
GSM8K	70%	82%	+12 pp
IFEval	68%	71%	+3 pp
HumanEval	75%	80%	+5 pp

Publishing an MMLU-Pro result in 0-shot without saying so, when the leaderboard standard is 5-shot, inflates the comparison by ~10 pp. Two results from the same model published with different numbers of shots are not comparable.

Prompt sensitivity

Minor perturbations in the prompt template produce variations of up to ±8 pp on MCQ and ±15 pp on generation tasks. Sources of variance identified in the literature:

Order of the options in MCQ (position bias): up to ±4 pp.
Wording of the question (paraphrasing): ±3–8 pp.
Presence or absence of “Let’s think step by step”: ±5–12 pp on reasoning.
Language of the system instructions: ±3–10 pp in non-multilingual models.

Why two evals give different results

A common scenario: the paper reports MMLU-Pro 74.3% and the reproduction gives 71.8%. Frequent causes:

Cause	Typical magnitude
Different harness version (changed template)	1–4 pp
Model with a silent update	0.5–3 pp
Different context truncation	0.5–2 pp
Sampling seed (for generation)	0.3–1.5 pp
Tokenisation with a chat template vs without one	2–8 pp

The sum of these sources explains gaps of 2–6 pp that are not noise but systematic protocol differences.

The confidence interval formula for comparing two models

To know whether the difference between model A (accuracy $p_A$) and model B ($p_B$) over a test set of size $n$ is statistically significant:

\[ \Delta p \pm 1.96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

Applied to GPQA Diamond ($n = 198$), if $p_A = 0.90$ and $p_B = 0.88$:

\[ \Delta p = 0.02 \quad;\quad \text{margin} = 1.96 \cdot \sqrt{\frac{0.09 + 0.1056}{198}} \approx 1.96 \cdot 0.0314 \approx 0.062 \]

The 95% confidence interval is $[{-0.042},\ {+0.082}]$: it crosses zero. A difference of 2 pp over 198 questions is not statistically distinguishable from noise. To distinguish 90% from 88% with 95% confidence you need:

\[ n \approx \frac{(1.96)^2 \cdot (p_A(1-p_A) + p_B(1-p_B))}{(\Delta p)^2} \approx \frac{3.84 \cdot 0.1956}{0.0004} \approx 1{,}878 \text{ questions} \]

GPQA Diamond is not that size. Differences of 1–2 pp on GPQA Diamond are statistically invisible.

The quality axis in the scorecard

The platform’s scorecard has three columns: performance (goodput under SLO), quality (score on a reference benchmark), energy (J/token). The quality column is built like this:

Field	Content
Primary benchmark	MMLU-Pro (5-shot) — separates in the 60–77% zone
Reasoning benchmark	GPQA Diamond (0-shot) — separates in the 55–90% zone
Instruction benchmark	IFEval (0-shot prompt-level accuracy)
Code benchmark	MBPP (0-shot, pass@1) — less saturated than HumanEval
Anti-contamination	LiveBench global score (monthly)
Tool	lm-evaluation-harness (primary) + lighteval (secondary, HF Hub)
Pinned version	`harness==0.4.3`, `lighteval==0.6.x`
Shots	Declared explicitly per benchmark
Reproducibility	Output JSON stored with the model SHA
Frequency	Per model release; not in the CI of every PR (GPU-hour cost)

A full eval over 4 benchmarks (MMLU-Pro 12,032 + GPQA 198 + IFEval 541 + MBPP 500) with a 70B model on a generic 2×H100 node takes on the order of 4–8 hours depending on batch size and number of shots. The GPU cost is real: at ~5 €/h per amortised card, a full eval costs on the order of 40–80 €. It is not run on every commit; it is run on every model release or major adapter release.

Sources

EleutherAI · lm-evaluation-harness — https://github.com/EleutherAI/lm-evaluation-harness
Stanford CRFM · HELM — https://github.com/stanford-crfm/helm
Stanford CRFM · HELM Capabilities (mar-2025) — https://crfm.stanford.edu/2025/03/20/helm-capabilities.html
Stanford CRFM · HELM (sitio oficial) — https://crfm.stanford.edu/helm/
LiveBench · paper arXiv:2406.19314 — https://arxiv.org/abs/2406.19314
LiveBench · sitio oficial — https://livebench.ai
LiveBench · GitHub — https://github.com/LiveBench/LiveBench
NYU Center for Data Science · LiveBench blog — https://nyudatascience.medium.com/livebench-challenging-language-models-with-contamination-free-questions-999b52967ec8
Hugging Face · lighteval — https://github.com/huggingface/lighteval
Hugging Face · OpenLLM Leaderboard (archivado) — https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
Hugging Face · OpenEvals (sucesor) — https://huggingface.co/spaces/OpenEvals/find-a-leaderboard
Hugging Face · Archived Open LLM Leaderboard 2024-2025 — https://huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025
LMArena · Chatbot Arena — https://lmarena.ai
LMSys · blog Chatbot Arena Elo — https://www.lmsys.org/blog/2023-05-03-arena/
“A Statistical Framework for Ranking LLM-Based Chatbots” · arXiv:2412.18407 — https://arxiv.org/pdf/2412.18407
“Data Contamination or Genuine Generalization?” (2025) — https://www.suaspress.org/ojs/index.php/AJNS/article/view/v2n2a03
“When Benchmarks Leak: Inference-Time Decontamination” (2025) · arXiv:2601.19334 — https://arxiv.org/pdf/2601.19334
“How Contaminated Is Your Benchmark?” (2025) · arXiv:2502.00678 — https://arxiv.org/pdf/2502.00678
“When AI Benchmarks Plateau” (2026) · arXiv:2602.16763 — https://arxiv.org/pdf/2602.16763
GPQA · arXiv:2311.12022 — https://arxiv.org/abs/2311.12022
GPQA Diamond leaderboard · Artificial Analysis — https://artificialanalysis.ai/evaluations/gpqa-diamond
“The Emperor’s New Clothes in Benchmarking?” (2025) · arXiv:2503.16402 — https://arxiv.org/pdf/2503.16402
lm-evaluation-harness · CLI reference — https://lm-evaluation-harness.readthedocs.io/running_evals/cli_reference/
crfm-helm · PyPI — https://pypi.org/project/crfm-helm/

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

Thu, 04 Jun 2026 09:00:00 +0200

TL;DR

A RAG pipeline fails in ways that user satisfaction cannot distinguish: the LLM can hallucinate even with good chunks, or retrieval can ignore key documents even though the LLM synthesises well what it receives. RAGAS breaks evaluation down into four orthogonal metrics, faithfulness, answer relevance, context precision and context recall, each pointing at a different sub-component. The golden dataset is the reference calibrator; without it the metrics have no anchor. The complete stack runs 100 % on-premise with vLLM as the judge and Langfuse for traceability.

The master analogy: the quality inspector at a furniture factory

Imagine you manufacture chairs. You could ask customers “is it comfortable?” and leave it there. But that question does not tell you what to fix when the answer is “no”. The quality inspector does not ask that: they measure the board with a Shore hardness test, check that each leg is exactly 45 cm, verify that the assembly manual includes the twelve screws from the BOM, and detect whether a low-density board slipped past the incoming filter.

RAGAS is that inspector applied to RAG:

Faithfulness → does the board have the specified hardness? The LLM can only use the material (chunks) that retrieval hands it.
Context Precision → is the leg exactly the right length? Of the K retrieved chunks, how many are genuinely useful and how many are filler that confuses the assembler?
Context Recall → does the manual include all the screws? Of all the facts the correct answer should contain, how many appear in the retrieved chunks?
Noise Sensitivity → if the operator uses a medium-low density board, does it show in the final product? If you introduce irrelevant chunks, does the LLM start hallucinating?

Without measuring each dimension separately, the diagnosis is opaque: “the RAG is not working well” does not tell you whether to repair the embedder, the reranker, the prompt or the corpus.

The problem of evaluating RAG

Classification has one uncomfortable virtue: if you predict 87 out of 100 labels correctly, accuracy = 0.87. There is no ambiguity. RAG does not enjoy that luxury.

A RAG system can fail in at least three independent ways:

Retrieval correct, LLM hallucinates: the chunks contain the correct answer, but the LLM generates claims that are not in those chunks. Faithfulness low; context recall high.
LLM correct, retrieval fails: retrieval returns irrelevant chunks (low context precision) or incomplete ones (low context recall). If the LLM has enough parametric knowledge, it may look as though it is answering well, but in reality it is ignoring the context, which is a time bomb once the parametric knowledge goes stale.
Retrieval and LLM both correct, answer does not answer the question: the answer is faithful to the context and the chunks are relevant, but the question was a different one. Answer relevance low.

Each failure mode requires a different metric and a different corrective action. Using a single metric (BLEU, ROUGE, user satisfaction) mixes the signals and makes it impossible to prioritise improvement work.

The four RAGAS metrics

1. Faithfulness — fidelity to the context

Question: how many claims in the generated answer are supported by the retrieved chunks?

Calculation:

$$\text{Faithfulness} = \frac{|\text{claims supported by the context}|}{|\text{total claims in the answer}|}$$

The process uses an LLM-as-judge (see https://blog.lo0.es/en/posts/llm-as-judge-exam-marker-not-oracle/): first the atomic claims are extracted from the answer (“the model was released in 2023”, “it supports contexts of 128k tokens”, and so on), then the judge classifies each claim as supported or not supported by the chunks.

Example: The generated answer has 5 claims. The judge determines that 4 are in the chunks and 1 is an unsupported extrapolation.

$$\text{Faithfulness} = \frac{4}{5} = 0{.}80$$

Warning sign: faithfulness < 0.85 indicates that the LLM is generating content that goes beyond the context, which is to say it is hallucinating with superficial backing.

2. Answer Relevance — relevance of the answer

Question: does the answer actually answer the question as posed?

Intuition: An answer that answers the question well “implies” that question. If you generate N hypothetical questions from the answer and measure their semantic similarity to the original question, you get a relevance signal.

Calculation:

$$\text{AnswerRelevance} = \frac{1}{N} \sum_{i=1}^{N} \cos(\vec{q}_{\text{original}}, \vec{q}_{i}^{\text{generated}})$$

where $\vec{q}$ are question embeddings.

Example: For the question “Which Python versions does FastAPI support?” and an answer about web frameworks in general, the hypothetical questions generated will be along the lines of “what are the best web frameworks?”, with low cosine against the original question → low answer relevance.

3. Context Precision — retrieval precision

Question: of the K retrieved chunks, what proportion are genuinely relevant?

Calculation (weighted version):

$$\text{ContextPrecision@K} = \frac{\sum_{k=1}^{K} \text{Precision@}k \cdot \mathbb{1}[\text{chunk}_k \text{ is relevant}]}{|\text{relevant chunks in top-K}|}$$

The most direct form: the LLM judge classifies each chunk as relevant or not for answering the question. Precision is the relevant fraction.

Example: 5 chunks are retrieved. The judge considers 3 relevant and 2 noise.

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

Warning sign: precision < 0.6 indicates that retrieval is contaminating the context with information that can contradict or dilute the correct answer.

4. Context Recall — retrieval recall

Question: of all the facts needed to build the correct answer (ground truth), what proportion are covered by the retrieved chunks?

Calculation:

$$\text{ContextRecall} = \frac{|\text{ground-truth claims attributable to some chunk}|}{|\text{total claims in ground truth}|}$$

This metric requires ground truth, which is to say you need the golden dataset.

Example: The ground truth has 6 claims. The retrieved chunks cover 5 of them; the sixth is in a document that retrieval did not find.

$$\text{ContextRecall} = \frac{5}{6} \approx 0{.}83$$

5. Noise Sensitivity — sensitivity to noise

Question: if you introduce irrelevant chunks into the context, how far does faithfulness fall?

It is computed as the difference in faithfulness between the clean condition and the condition with controlled noise:

$$\Delta\text{Faithfulness} = \text{Faithfulness}_{\text{clean}} - \text{Faithfulness}_{\text{noisy}}$$

A $\Delta > 0{.}15$ indicates that the prompt or the LLM are fragile against contaminated contexts, a problem that low context precision turns into a routine occurrence.

A complete worked example

Scenario

Question: “What latency guarantees does Redis offer for in-memory read operations with a 10 GB dataset?”

Answer generated by the RAG:

“Redis guarantees sub-millisecond latencies for GET operations on datasets that fit in RAM. With a 10 GB dataset and clients on the same local network, P99 is typically below 1 ms. Redis also supports horizontal clustering to scale beyond the RAM of a single node. In addition, Redis has included native JSON support since version 7.2.”

Ground truth (reference answer):

“Redis operates entirely in memory, which guarantees sub-millisecond latencies for GET. On local networks with 10 GB datasets in RAM, P99 sits below 1 ms. Clustering allows scaling beyond the RAM of a single node.”

Retrieved chunks (5 chunks, summarised fragments):

#	Summarised content	Relevant
C1	“Redis operates in memory; GET has latencies < 1 ms on LAN”	Yes
C2	“Redis Cluster allows sharding to scale total RAM”	Yes
C3	“Redis Sentinel manages high availability through automatic failover”	No
C4	“Redis benchmarks: P50 = 0.3 ms, P99 = 0.9 ms on a 10 GB dataset”	Yes
C5	“Redis Stack adds modules: RedisJSON, RediSearch, RedisTimeSeries”	No

Step-by-step calculation

Faithfulness:

Claims in the generated answer:

“Redis guarantees sub-millisecond latencies for GET on in-RAM datasets” → supported by C1, C4
“With 10 GB on LAN, P99 < 1 ms” → supported by C4
“Redis supports horizontal clustering to scale RAM” → supported by C2
“Redis has included native JSON support since version 7.2” → NOT supported by any chunk (C5 mentions RedisJSON as a Redis Stack module, not as native to Redis core)

$$\text{Faithfulness} = \frac{3}{4} = 0{.}75$$

Claim 4 is an extrapolation that mixes information from C5 imprecisely, a partial hallucination.

Context Precision:

Relevant chunks: C1, C2, C4 (3 out of 5).

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

C3 and C5 are noise. C5 in particular contributed to the partial hallucination about JSON.

Context Recall:

Ground-truth claims:

“Redis operates in memory, GET < 1 ms” → attributable to C1 ✓
“P99 < 1 ms on LAN with 10 GB” → attributable to C4 ✓
“Clustering scales beyond the RAM of a single node” → attributable to C2 ✓

$$\text{ContextRecall} = \frac{3}{3} = 1{.}00$$

Retrieval found every chunk needed for the ground truth. The problem is not recall but precision (C3 and C5 contaminated the context).

Answer Relevance:

The judge generates 3 hypothetical questions from the answer:

“What latencies does Redis offer for in-memory reads?” — cos = 0.91
“How does Redis scale horizontally?” — cos = 0.74
“Which JSON modules does Redis include?” — cos = 0.52 (a drift caused by the hallucination)

$$\text{AnswerRelevance} = \frac{0{.}91 + 0{.}74 + 0{.}52}{3} = 0{.}72$$

The drift towards JSON reduced relevance. A tighter answer would have scored around 0.90.

Summary of the example

Metric	Value	Diagnosis
Faithfulness	0.75	LLM extrapolated beyond the context
Context Precision	0.60	Retrieval returned 2 irrelevant chunks
Context Recall	1.00	Retrieval captured everything needed
Answer Relevance	0.72	Answer drifts off topic

Main corrective action: improve the reranker so that C3 and C5 are filtered before they reach the LLM. The faithfulness and relevance problem is a direct consequence of the low precision, not of the LLM itself.

Building the golden dataset

What it is and why it matters

The golden dataset is a set of (question, relevant chunks, correct answer) tuples that acts as a reference calibrator. Without it, context recall cannot be computed (there is no ground truth) and the other metrics lack any interpretive anchor: is 0.75 faithfulness good or bad for this corpus and this domain?

A well-built golden dataset lets you:

Compare pipeline versions (embedder v1 vs v2, chunk size 512 vs 1024)
Detect regressions in CI before deploying
Stratify the analysis by question type

LLM-assisted construction pipeline

Purely manual construction is expensive. The standard pattern in 2026 is LLM assistance with human review of a sample:

Step 1 — Seed chunk selection. From the full corpus, select representative chunks through stratified sampling (by section, date, document type). For a technical corpus of 10,000 chunks, 500-1,000 seeds is a reasonable starting point.

Step 2 — Question generation. A powerful LLM (Llama-3.1-70B or similar) generates 2-3 questions per seed chunk using a prompt along these lines:

Given the following documentation fragment, generate specific questions
that can only be answered correctly using THIS fragment and not
general knowledge. The questions should be the ones an engineer looking
for operational information would ask.
Fragment: {chunk}

Step 3 — Reference answer generation. The same LLM, with access to the seed chunk (and to adjacent chunks if the question requires them), generates the reference answer.

Step 4 — Human review of a sample. Manually review 10-20 % of the generated dataset. The most common rejection criteria: trivial questions that any LLM answers without the corpus, answers the LLM padded with parametric knowledge instead of the chunks, and badly formulated or ambiguous questions.

Minimum size

Use case	Minimum pairs	Notes
Prototype / initial validation	50-100	Enough to spot coarse problems
Technical corpus in production	200-500	Allows basic stratification
Robust production with full stratification	500-1,000+	Needed to detect subtle regressions

Stratifying the dataset

A flat golden dataset measures the average but hides the extreme cases. The minimum recommended stratification includes three question types:

Easy (single-hop): A single chunk contains all the information needed. The baseline that any decent RAG must beat.
Hard (multi-hop): The correct answer requires combining information from 2-4 different chunks. This is where the limits of the reranker and the synthesis prompt show up.
Adversarial: The question has a false premise, or the corpus does not contain the answer. A correct RAG must answer “I do not have enough information”, whereas a fragile RAG hallucinates confidently. This question type directly measures the risk of high-impact hallucination.

The Goodhart trap

“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart

If you optimise the embedder or the reranker using the golden dataset as a loss function, the dataset is corrupted as a metric: the system learns to perform well on those specific questions without improving on the general domain.

The solution is the same as in supervised ML: separate the dev set (for optimisation and iteration) from the test set (for final evaluation, frozen and audited). The test set must never be used to make design decisions; only to report the state of the system at release time.

Correlation with real satisfaction

Field studies published by the Databricks teams (2024) and the RAGAS adoption analyses (2025) point to interpretable operational thresholds:

Metric range	Observable symptom	Corrective action
Faithfulness < 0.75	Users frequently report “made-up answers”	Review the LLM prompt; increase citation instructions; lower the temperature
Faithfulness 0.75-0.85	Occasional hallucinations on peripheral topics	Improve context precision to eliminate contaminating chunks
Faithfulness ≥ 0.85	Correlates with positive NPS in field studies	Hold steady; monitor for drift
Context Precision < 0.60	LLM includes contradictory information; inconsistent answers	Tune the reranker; reduce K; review similarity thresholds
Context Recall < 0.70	Multi-hop questions fail; key information missing	Review the chunking strategy; add larger chunks; enrich metadata
Answer Relevance < 0.70	Answers that are “correct but do not answer”	Review the synthesis prompt; add an explicit instruction to stick to the question

Low context precision is especially pernicious: irrelevant chunks are not neutral. They raise the probability that the LLM uses incorrect information as though it were relevant, degrading faithfulness in a chain reaction. It is the transmission belt by which a retrieval problem turns into an LLM problem.

Diagram: the continuous evaluation loop

The continuous evaluation loop: corpus → retrieval → LLM → RAGAS judge → metrics → alert → corrective action → corpus.

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

The ragas library supports asynchronous evaluation and multiple LLM backends. Integrating vLLM as the judge removes any need to send data to external APIs, which is critical in environments with sensitive data.

from ragas import evaluate
from ragas.metrics import (
 faithfulness,
 answer_relevancy,
 context_precision,
 context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Judge LLM pointing at on-premise vLLM
judge_llm = ChatOpenAI(
 model="meta-llama/Llama-3.1-70B-Instruct",
 base_url="http://vllm-service:8000/v1",
 api_key="sk-local", # vLLM ignores the value but requires the field
)

embeddings = OpenAIEmbeddings(
 model="BAAI/bge-m3",
 base_url="http://embedding-service:8001/v1",
 api_key="sk-local",
)

result = evaluate(
 dataset=golden_dataset, # HuggingFace Dataset with standard columns
 metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
 llm=judge_llm,
 embeddings=embeddings,
)

The dataset RAGAS expects has four columns: question, answer, contexts (a list of strings), ground_truth.

Langfuse for eval traceability

Each RAGAS evaluation is recorded in Langfuse as a dataset experiment, linking the scores to the production spans (see https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/). This makes it possible to correlate a drop in faithfulness with the specific request that caused it. Without that link, the metrics are numbers with no actionable context.

from langfuse import Langfuse

lf = Langfuse()

# Create or retrieve the dataset in Langfuse
dataset = lf.get_or_create_dataset("rag-golden-v3")

# Record the experiment scores
for idx, row in result.to_pandas().iterrows():
 lf.score(
 name="ragas-faithfulness",
 value=row["faithfulness"],
 trace_id=row["trace_id"], # linked to the production span
 )

Prometheus + Grafana for operational alerts

RAGAS metrics are exposed as Prometheus gauges. A Grafana dashboard with thresholds configures alerts when faithfulness falls sustainedly below 0.80:

# Prometheus alert rule
- alert: RAGFaithfulnessLow
 expr: avg_over_time(rag_faithfulness_score[30m]) < 0.80
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "RAG faithfulness below threshold ({{ $value | humanize }})"
 description: "Review context precision and the reranker. Possible corpus drift."

Running RAGAS against on-premise vLLM — practical considerations

Judge size: Llama-3.1-70B as the judge produces results comparable to GPT-4 on faithfulness and context evaluation, according to the RAGAS 0.2 benchmarks (2025). Smaller models (8B-13B) degrade judge quality on multi-hop questions.
Throughput: On on-premise hardware with 4×H100 SXM (320 GB, NVLink), a run of 200 evaluations with Llama-3.1-70B takes roughly 8-12 minutes with batch_size=8 and vLLM in continuous batching mode.
Cost per evaluation: With no external API, the marginal cost is electricity plus GPU amortisation. With 4×H100 at around 3 kW sustained, a run of 200 evaluations costs < 0.10 € in energy at a typical industrial tariff.
Recommended frequency: weekly offline eval over the full golden dataset plus sampled online eval (5-10 % of production requests) with a subset of metrics that do not require ground truth (faithfulness, answer relevance).

What we have not covered

Alternatives to RAGAS: TruLens (evaluation with modular feedbacks), DeepEval (programmatic assertions, pytest integration), ARES (Stanford framework with trained classifiers instead of LLM-as-judge), and the OpenAI evals framework. Each has different trade-offs in judge cost, reliability and ease of integration.
Continuous eval in production: automatically sampling real requests, anonymising them, running a subset of metrics without ground truth and using the result to detect system drift before users report it. It requires a data pipeline separate from the inference pipeline.
Multilingual eval: RAGAS with a judge in Spanish or Catalan over a non-English corpus has documented biases when the judge is a model trained fundamentally in English. The semantic similarity embeddings for answer relevance are especially sensitive to corpus language vs judge language.
A/B testing of RAG configurations: using RAGAS metrics as the success criterion in controlled experiments, chunk size 512 vs 1024, pure BM25 vs hybrid, cross-encoder reranker vs biencoder, with statistical significance computed over the golden dataset.

References

Es Shahul, et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 (2023). https://arxiv.org/abs/2309.15217
RAGAS Documentation v0.2. Metrics Reference. https://docs.ragas.io/en/stable/concepts/metrics/ (accessed June 2026)
Langfuse. Dataset Experiments. https://langfuse.com/docs/datasets/overview (accessed June 2026)
Databricks. LLM Quality Evaluation: From Lab to Production. Databricks Engineering Blog (2024).
Saad-Falcon, J. et al. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476 (2023).
Goodhart, C.A.E. Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics. Reserve Bank of Australia (1975). Modern formulation of the law that bears his name.
vLLM Project. OpenAI-Compatible Server. https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html (accessed June 2026)

Evaluation on lo0 — Tech Blog

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

TL;DR

Quality ≠ performance: the independent Pareto axis

Reference benchmarks: what each one measures

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

HELM (Stanford CRFM)

LiveBench

lighteval (Hugging Face)

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

LMArena / Chatbot Arena

Decision table: which tool for which objective

The data contamination trap

What it is

Scale of the problem (2024–2025 data)

Detection mechanisms

Why static leaderboards saturate

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

Few-shot vs zero-shot: the numerical difference

Prompt sensitivity

Why two evals give different results

The confidence interval formula for comparing two models

The quality axis in the scorecard

See also

Sources

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

TL;DR

The master analogy: the quality inspector at a furniture factory

The problem of evaluating RAG

The four RAGAS metrics

1. Faithfulness — fidelity to the context

2. Answer Relevance — relevance of the answer

3. Context Precision — retrieval precision

4. Context Recall — retrieval recall

5. Noise Sensitivity — sensitivity to noise

A complete worked example

Scenario

Step-by-step calculation

Summary of the example

Building the golden dataset

What it is and why it matters

LLM-assisted construction pipeline

Minimum size

Stratifying the dataset

The Goodhart trap

Correlation with real satisfaction

Diagram: the continuous evaluation loop

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

Langfuse for eval traceability

Prometheus + Grafana for operational alerts

Running RAGAS against on-premise vLLM — practical considerations

What we have not covered

See also

References