Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

TL;DR

A RAG pipeline fails in ways that user satisfaction cannot distinguish: the LLM can hallucinate even with good chunks, or retrieval can ignore key documents even though the LLM synthesises well what it receives. RAGAS breaks evaluation down into four orthogonal metrics, faithfulness, answer relevance, context precision and context recall, each pointing at a different sub-component. The golden dataset is the reference calibrator; without it the metrics have no anchor. The complete stack runs 100 % on-premise with vLLM as the judge and Langfuse for traceability.

The master analogy: the quality inspector at a furniture factory

Imagine you manufacture chairs. You could ask customers “is it comfortable?” and leave it there. But that question does not tell you what to fix when the answer is “no”. The quality inspector does not ask that: they measure the board with a Shore hardness test, check that each leg is exactly 45 cm, verify that the assembly manual includes the twelve screws from the BOM, and detect whether a low-density board slipped past the incoming filter.

RAGAS is that inspector applied to RAG:

• Faithfulness → does the board have the specified hardness? The LLM can only use the material (chunks) that retrieval hands it.
• Context Precision → is the leg exactly the right length? Of the K retrieved chunks, how many are genuinely useful and how many are filler that confuses the assembler?
• Context Recall → does the manual include all the screws? Of all the facts the correct answer should contain, how many appear in the retrieved chunks?
• Noise Sensitivity → if the operator uses a medium-low density board, does it show in the final product? If you introduce irrelevant chunks, does the LLM start hallucinating?

Without measuring each dimension separately, the diagnosis is opaque: “the RAG is not working well” does not tell you whether to repair the embedder, the reranker, the prompt or the corpus.

The problem of evaluating RAG

Classification has one uncomfortable virtue: if you predict 87 out of 100 labels correctly, accuracy = 0.87. There is no ambiguity. RAG does not enjoy that luxury.

A RAG system can fail in at least three independent ways:

1. Retrieval correct, LLM hallucinates: the chunks contain the correct answer, but the LLM generates claims that are not in those chunks. Faithfulness low; context recall high.
2. LLM correct, retrieval fails: retrieval returns irrelevant chunks (low context precision) or incomplete ones (low context recall). If the LLM has enough parametric knowledge, it may look as though it is answering well, but in reality it is ignoring the context, which is a time bomb once the parametric knowledge goes stale.
3. Retrieval and LLM both correct, answer does not answer the question: the answer is faithful to the context and the chunks are relevant, but the question was a different one. Answer relevance low.

Each failure mode requires a different metric and a different corrective action. Using a single metric (BLEU, ROUGE, user satisfaction) mixes the signals and makes it impossible to prioritise improvement work.

The four RAGAS metrics

1. Faithfulness — fidelity to the context

Question: how many claims in the generated answer are supported by the retrieved chunks?

Calculation:

The process uses an LLM-as-judge (see https://blog.lo0.es/en/posts/llm-as-judge-exam-marker-not-oracle/): first the atomic claims are extracted from the answer (“the model was released in 2023”, “it supports contexts of 128k tokens”, and so on), then the judge classifies each claim as supported or not supported by the chunks.

Example: The generated answer has 5 claims. The judge determines that 4 are in the chunks and 1 is an unsupported extrapolation.

Warning sign: faithfulness < 0.85 indicates that the LLM is generating content that goes beyond the context, which is to say it is hallucinating with superficial backing.

2. Answer Relevance — relevance of the answer

Question: does the answer actually answer the question as posed?

Intuition: An answer that answers the question well “implies” that question. If you generate N hypothetical questions from the answer and measure their semantic similarity to the original question, you get a relevance signal.

Calculation:

where $\vec{q}$ are question embeddings.

Example: For the question “Which Python versions does FastAPI support?” and an answer about web frameworks in general, the hypothetical questions generated will be along the lines of “what are the best web frameworks?”, with low cosine against the original question → low answer relevance.

3. Context Precision — retrieval precision

Question: of the K retrieved chunks, what proportion are genuinely relevant?

Calculation (weighted version):

The most direct form: the LLM judge classifies each chunk as relevant or not for answering the question. Precision is the relevant fraction.

Example: 5 chunks are retrieved. The judge considers 3 relevant and 2 noise.

Warning sign: precision < 0.6 indicates that retrieval is contaminating the context with information that can contradict or dilute the correct answer.

4. Context Recall — retrieval recall

Question: of all the facts needed to build the correct answer (ground truth), what proportion are covered by the retrieved chunks?

Calculation:

This metric requires ground truth, which is to say you need the golden dataset.

Example: The ground truth has 6 claims. The retrieved chunks cover 5 of them; the sixth is in a document that retrieval did not find.

5. Noise Sensitivity — sensitivity to noise

Question: if you introduce irrelevant chunks into the context, how far does faithfulness fall?

It is computed as the difference in faithfulness between the clean condition and the condition with controlled noise:

A $\Delta > 0{.}15$ indicates that the prompt or the LLM are fragile against contaminated contexts, a problem that low context precision turns into a routine occurrence.

A complete worked example

Scenario

Question: “What latency guarantees does Redis offer for in-memory read operations with a 10 GB dataset?”

Answer generated by the RAG:

“Redis guarantees sub-millisecond latencies for GET operations on datasets that fit in RAM. With a 10 GB dataset and clients on the same local network, P99 is typically below 1 ms. Redis also supports horizontal clustering to scale beyond the RAM of a single node. In addition, Redis has included native JSON support since version 7.2.”

Ground truth (reference answer):

“Redis operates entirely in memory, which guarantees sub-millisecond latencies for GET. On local networks with 10 GB datasets in RAM, P99 sits below 1 ms. Clustering allows scaling beyond the RAM of a single node.”

Retrieved chunks (5 chunks, summarised fragments):

Step-by-step calculation

Faithfulness:

Claims in the generated answer:

1. “Redis guarantees sub-millisecond latencies for GET on in-RAM datasets” → supported by C1, C4
2. “With 10 GB on LAN, P99 < 1 ms” → supported by C4
3. “Redis supports horizontal clustering to scale RAM” → supported by C2
4. “Redis has included native JSON support since version 7.2” → NOT supported by any chunk (C5 mentions RedisJSON as a Redis Stack module, not as native to Redis core)

Claim 4 is an extrapolation that mixes information from C5 imprecisely, a partial hallucination.

Context Precision:

Relevant chunks: C1, C2, C4 (3 out of 5).

C3 and C5 are noise. C5 in particular contributed to the partial hallucination about JSON.

Context Recall:

Ground-truth claims:

1. “Redis operates in memory, GET < 1 ms” → attributable to C1 ✓
2. “P99 < 1 ms on LAN with 10 GB” → attributable to C4 ✓
3. “Clustering scales beyond the RAM of a single node” → attributable to C2 ✓

Retrieval found every chunk needed for the ground truth. The problem is not recall but precision (C3 and C5 contaminated the context).

Answer Relevance:

The judge generates 3 hypothetical questions from the answer:

• “What latencies does Redis offer for in-memory reads?” — cos = 0.91
• “How does Redis scale horizontally?” — cos = 0.74
• “Which JSON modules does Redis include?” — cos = 0.52 (a drift caused by the hallucination)

The drift towards JSON reduced relevance. A tighter answer would have scored around 0.90.

Summary of the example

Main corrective action: improve the reranker so that C3 and C5 are filtered before they reach the LLM. The faithfulness and relevance problem is a direct consequence of the low precision, not of the LLM itself.

Building the golden dataset

What it is and why it matters

The golden dataset is a set of (question, relevant chunks, correct answer) tuples that acts as a reference calibrator. Without it, context recall cannot be computed (there is no ground truth) and the other metrics lack any interpretive anchor: is 0.75 faithfulness good or bad for this corpus and this domain?

A well-built golden dataset lets you:

• Compare pipeline versions (embedder v1 vs v2, chunk size 512 vs 1024)
• Detect regressions in CI before deploying
• Stratify the analysis by question type

LLM-assisted construction pipeline

Purely manual construction is expensive. The standard pattern in 2026 is LLM assistance with human review of a sample:

Step 1 — Seed chunk selection. From the full corpus, select representative chunks through stratified sampling (by section, date, document type). For a technical corpus of 10,000 chunks, 500-1,000 seeds is a reasonable starting point.

Step 2 — Question generation. A powerful LLM (Llama-3.1-70B or similar) generates 2-3 questions per seed chunk using a prompt along these lines:

Given the following documentation fragment, generate specific questions
that can only be answered correctly using THIS fragment and not
general knowledge. The questions should be the ones an engineer looking
for operational information would ask.

Fragment: {chunk}

Step 3 — Reference answer generation. The same LLM, with access to the seed chunk (and to adjacent chunks if the question requires them), generates the reference answer.

Step 4 — Human review of a sample. Manually review 10-20 % of the generated dataset. The most common rejection criteria: trivial questions that any LLM answers without the corpus, answers the LLM padded with parametric knowledge instead of the chunks, and badly formulated or ambiguous questions.

Minimum size

Stratifying the dataset

A flat golden dataset measures the average but hides the extreme cases. The minimum recommended stratification includes three question types:

• Easy (single-hop): A single chunk contains all the information needed. The baseline that any decent RAG must beat.
• Hard (multi-hop): The correct answer requires combining information from 2-4 different chunks. This is where the limits of the reranker and the synthesis prompt show up.
• Adversarial: The question has a false premise, or the corpus does not contain the answer. A correct RAG must answer “I do not have enough information”, whereas a fragile RAG hallucinates confidently. This question type directly measures the risk of high-impact hallucination.

The Goodhart trap

“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart

If you optimise the embedder or the reranker using the golden dataset as a loss function, the dataset is corrupted as a metric: the system learns to perform well on those specific questions without improving on the general domain.

The solution is the same as in supervised ML: separate the dev set (for optimisation and iteration) from the test set (for final evaluation, frozen and audited). The test set must never be used to make design decisions; only to report the state of the system at release time.

Correlation with real satisfaction

Field studies published by the Databricks teams (2024) and the RAGAS adoption analyses (2025) point to interpretable operational thresholds:

Low context precision is especially pernicious: irrelevant chunks are not neutral. They raise the probability that the LLM uses incorrect information as though it were relevant, degrading faithfulness in a chain reaction. It is the transmission belt by which a retrieval problem turns into an LLM problem.

Diagram: the continuous evaluation loop

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

The ragas library supports asynchronous evaluation and multiple LLM backends. Integrating vLLM as the judge removes any need to send data to external APIs, which is critical in environments with sensitive data.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Judge LLM pointing at on-premise vLLM
judge_llm = ChatOpenAI(
    model="meta-llama/Llama-3.1-70B-Instruct",
    base_url="http://vllm-service:8000/v1",
    api_key="sk-local",  # vLLM ignores the value but requires the field
)

embeddings = OpenAIEmbeddings(
    model="BAAI/bge-m3",
    base_url="http://embedding-service:8001/v1",
    api_key="sk-local",
)

result = evaluate(
    dataset=golden_dataset,  # HuggingFace Dataset with standard columns
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    llm=judge_llm,
    embeddings=embeddings,
)

The dataset RAGAS expects has four columns: question, answer, contexts (a list of strings), ground_truth.

Langfuse for eval traceability

Each RAGAS evaluation is recorded in Langfuse as a dataset experiment, linking the scores to the production spans (see https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/). This makes it possible to correlate a drop in faithfulness with the specific request that caused it. Without that link, the metrics are numbers with no actionable context.

from langfuse import Langfuse

lf = Langfuse()

# Create or retrieve the dataset in Langfuse
dataset = lf.get_or_create_dataset("rag-golden-v3")

# Record the experiment scores
for idx, row in result.to_pandas().iterrows():
    lf.score(
        name="ragas-faithfulness",
        value=row["faithfulness"],
        trace_id=row["trace_id"],  # linked to the production span
    )

Prometheus + Grafana for operational alerts

RAGAS metrics are exposed as Prometheus gauges. A Grafana dashboard with thresholds configures alerts when faithfulness falls sustainedly below 0.80:

# Prometheus alert rule
- alert: RAGFaithfulnessLow
  expr: avg_over_time(rag_faithfulness_score[30m]) < 0.80
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "RAG faithfulness below threshold ({{ $value | humanize }})"
    description: "Review context precision and the reranker. Possible corpus drift."

Running RAGAS against on-premise vLLM — practical considerations

• Judge size: Llama-3.1-70B as the judge produces results comparable to GPT-4 on faithfulness and context evaluation, according to the RAGAS 0.2 benchmarks (2025). Smaller models (8B-13B) degrade judge quality on multi-hop questions.
• Throughput: On on-premise hardware with 4×H100 SXM (320 GB, NVLink), a run of 200 evaluations with Llama-3.1-70B takes roughly 8-12 minutes with batch_size=8 and vLLM in continuous batching mode.
• Cost per evaluation: With no external API, the marginal cost is electricity plus GPU amortisation. With 4×H100 at around 3 kW sustained, a run of 200 evaluations costs < 0.10 € in energy at a typical industrial tariff.
• Recommended frequency: weekly offline eval over the full golden dataset plus sampled online eval (5-10 % of production requests) with a subset of metrics that do not require ground truth (faithfulness, answer relevance).

What we have not covered

• Alternatives to RAGAS: TruLens (evaluation with modular feedbacks), DeepEval (programmatic assertions, pytest integration), ARES (Stanford framework with trained classifiers instead of LLM-as-judge), and the OpenAI evals framework. Each has different trade-offs in judge cost, reliability and ease of integration.
• Continuous eval in production: automatically sampling real requests, anonymising them, running a subset of metrics without ground truth and using the result to detect system drift before users report it. It requires a data pipeline separate from the inference pipeline.
• Multilingual eval: RAGAS with a judge in Spanish or Catalan over a non-English corpus has documented biases when the judge is a model trained fundamentally in English. The semantic similarity embeddings for answer relevance are especially sensitive to corpus language vs judge language.
• A/B testing of RAG configurations: using RAGAS metrics as the success criterion in controlled experiments, chunk size 512 vs 1024, pure BM25 vs hybrid, cross-encoder reranker vs biencoder, with statistical significance computed over the golden dataset.

See also

• https://blog.lo0.es/en/posts/llm-as-judge-exam-marker-not-oracle/ — the LLM judge pattern that RAGAS uses to measure faithfulness claim by claim
• https://blog.lo0.es/en/posts/evals-llm-layer-after-tracing/ — the general evals framework in which RAGAS is the RAG specialisation
• https://blog.lo0.es/en/posts/reranker-hybrid-retrieval-five-chunks-llm-reads/ — the retrieval layer whose context precision and recall these metrics measure
• https://blog.lo0.es/en/posts/rag-corpus-curation-active-librarian/ — the corpus quality that context recall reflects
• https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/ — the production spans where Langfuse records the RAGAS scores
• https://blog.lo0.es/en/posts/data-versioning-llmops-dvc-lakefs-reproducible-golden-dataset/ — the golden dataset is a data artefact that needs versioning just like the corpus

References

1. Es Shahul, et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 (2023). https://arxiv.org/abs/2309.15217
2. RAGAS Documentation v0.2. Metrics Reference. https://docs.ragas.io/en/stable/concepts/metrics/ (accessed June 2026)
3. Langfuse. Dataset Experiments. https://langfuse.com/docs/datasets/overview (accessed June 2026)
4. Databricks. LLM Quality Evaluation: From Lab to Production. Databricks Engineering Blog (2024).
5. Saad-Falcon, J. et al. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476 (2023).
6. Goodhart, C.A.E. Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics. Reserve Bank of Australia (1975). Modern formulation of the law that bears his name.
7. vLLM Project. OpenAI-Compatible Server. https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html (accessed June 2026)