Golden-Dataset on lo0 — Tech Blog

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

Thu, 04 Jun 2026 09:00:00 +0200

TL;DR

A RAG pipeline fails in ways that user satisfaction cannot distinguish: the LLM can hallucinate even with good chunks, or retrieval can ignore key documents even though the LLM synthesises well what it receives. RAGAS breaks evaluation down into four orthogonal metrics, faithfulness, answer relevance, context precision and context recall, each pointing at a different sub-component. The golden dataset is the reference calibrator; without it the metrics have no anchor. The complete stack runs 100 % on-premise with vLLM as the judge and Langfuse for traceability.

The master analogy: the quality inspector at a furniture factory

Imagine you manufacture chairs. You could ask customers “is it comfortable?” and leave it there. But that question does not tell you what to fix when the answer is “no”. The quality inspector does not ask that: they measure the board with a Shore hardness test, check that each leg is exactly 45 cm, verify that the assembly manual includes the twelve screws from the BOM, and detect whether a low-density board slipped past the incoming filter.

RAGAS is that inspector applied to RAG:

Faithfulness → does the board have the specified hardness? The LLM can only use the material (chunks) that retrieval hands it.
Context Precision → is the leg exactly the right length? Of the K retrieved chunks, how many are genuinely useful and how many are filler that confuses the assembler?
Context Recall → does the manual include all the screws? Of all the facts the correct answer should contain, how many appear in the retrieved chunks?
Noise Sensitivity → if the operator uses a medium-low density board, does it show in the final product? If you introduce irrelevant chunks, does the LLM start hallucinating?

Without measuring each dimension separately, the diagnosis is opaque: “the RAG is not working well” does not tell you whether to repair the embedder, the reranker, the prompt or the corpus.

The problem of evaluating RAG

Classification has one uncomfortable virtue: if you predict 87 out of 100 labels correctly, accuracy = 0.87. There is no ambiguity. RAG does not enjoy that luxury.

A RAG system can fail in at least three independent ways:

Retrieval correct, LLM hallucinates: the chunks contain the correct answer, but the LLM generates claims that are not in those chunks. Faithfulness low; context recall high.
LLM correct, retrieval fails: retrieval returns irrelevant chunks (low context precision) or incomplete ones (low context recall). If the LLM has enough parametric knowledge, it may look as though it is answering well, but in reality it is ignoring the context, which is a time bomb once the parametric knowledge goes stale.
Retrieval and LLM both correct, answer does not answer the question: the answer is faithful to the context and the chunks are relevant, but the question was a different one. Answer relevance low.

Each failure mode requires a different metric and a different corrective action. Using a single metric (BLEU, ROUGE, user satisfaction) mixes the signals and makes it impossible to prioritise improvement work.

The four RAGAS metrics

1. Faithfulness — fidelity to the context

Question: how many claims in the generated answer are supported by the retrieved chunks?

Calculation:

$$\text{Faithfulness} = \frac{|\text{claims supported by the context}|}{|\text{total claims in the answer}|}$$

The process uses an LLM-as-judge (see https://blog.lo0.es/en/posts/llm-as-judge-exam-marker-not-oracle/): first the atomic claims are extracted from the answer (“the model was released in 2023”, “it supports contexts of 128k tokens”, and so on), then the judge classifies each claim as supported or not supported by the chunks.

Example: The generated answer has 5 claims. The judge determines that 4 are in the chunks and 1 is an unsupported extrapolation.

$$\text{Faithfulness} = \frac{4}{5} = 0{.}80$$

Warning sign: faithfulness < 0.85 indicates that the LLM is generating content that goes beyond the context, which is to say it is hallucinating with superficial backing.

2. Answer Relevance — relevance of the answer

Question: does the answer actually answer the question as posed?

Intuition: An answer that answers the question well “implies” that question. If you generate N hypothetical questions from the answer and measure their semantic similarity to the original question, you get a relevance signal.

Calculation:

$$\text{AnswerRelevance} = \frac{1}{N} \sum_{i=1}^{N} \cos(\vec{q}_{\text{original}}, \vec{q}_{i}^{\text{generated}})$$

where $\vec{q}$ are question embeddings.

Example: For the question “Which Python versions does FastAPI support?” and an answer about web frameworks in general, the hypothetical questions generated will be along the lines of “what are the best web frameworks?”, with low cosine against the original question → low answer relevance.

3. Context Precision — retrieval precision

Question: of the K retrieved chunks, what proportion are genuinely relevant?

Calculation (weighted version):

$$\text{ContextPrecision@K} = \frac{\sum_{k=1}^{K} \text{Precision@}k \cdot \mathbb{1}[\text{chunk}_k \text{ is relevant}]}{|\text{relevant chunks in top-K}|}$$

The most direct form: the LLM judge classifies each chunk as relevant or not for answering the question. Precision is the relevant fraction.

Example: 5 chunks are retrieved. The judge considers 3 relevant and 2 noise.

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

Warning sign: precision < 0.6 indicates that retrieval is contaminating the context with information that can contradict or dilute the correct answer.

4. Context Recall — retrieval recall

Question: of all the facts needed to build the correct answer (ground truth), what proportion are covered by the retrieved chunks?

Calculation:

$$\text{ContextRecall} = \frac{|\text{ground-truth claims attributable to some chunk}|}{|\text{total claims in ground truth}|}$$

This metric requires ground truth, which is to say you need the golden dataset.

Example: The ground truth has 6 claims. The retrieved chunks cover 5 of them; the sixth is in a document that retrieval did not find.

$$\text{ContextRecall} = \frac{5}{6} \approx 0{.}83$$

5. Noise Sensitivity — sensitivity to noise

Question: if you introduce irrelevant chunks into the context, how far does faithfulness fall?

It is computed as the difference in faithfulness between the clean condition and the condition with controlled noise:

$$\Delta\text{Faithfulness} = \text{Faithfulness}_{\text{clean}} - \text{Faithfulness}_{\text{noisy}}$$

A $\Delta > 0{.}15$ indicates that the prompt or the LLM are fragile against contaminated contexts, a problem that low context precision turns into a routine occurrence.

A complete worked example

Scenario

Question: “What latency guarantees does Redis offer for in-memory read operations with a 10 GB dataset?”

Answer generated by the RAG:

“Redis guarantees sub-millisecond latencies for GET operations on datasets that fit in RAM. With a 10 GB dataset and clients on the same local network, P99 is typically below 1 ms. Redis also supports horizontal clustering to scale beyond the RAM of a single node. In addition, Redis has included native JSON support since version 7.2.”

Ground truth (reference answer):

“Redis operates entirely in memory, which guarantees sub-millisecond latencies for GET. On local networks with 10 GB datasets in RAM, P99 sits below 1 ms. Clustering allows scaling beyond the RAM of a single node.”

Retrieved chunks (5 chunks, summarised fragments):

#	Summarised content	Relevant
C1	“Redis operates in memory; GET has latencies < 1 ms on LAN”	Yes
C2	“Redis Cluster allows sharding to scale total RAM”	Yes
C3	“Redis Sentinel manages high availability through automatic failover”	No
C4	“Redis benchmarks: P50 = 0.3 ms, P99 = 0.9 ms on a 10 GB dataset”	Yes
C5	“Redis Stack adds modules: RedisJSON, RediSearch, RedisTimeSeries”	No

Step-by-step calculation

Faithfulness:

Claims in the generated answer:

“Redis guarantees sub-millisecond latencies for GET on in-RAM datasets” → supported by C1, C4
“With 10 GB on LAN, P99 < 1 ms” → supported by C4
“Redis supports horizontal clustering to scale RAM” → supported by C2
“Redis has included native JSON support since version 7.2” → NOT supported by any chunk (C5 mentions RedisJSON as a Redis Stack module, not as native to Redis core)

$$\text{Faithfulness} = \frac{3}{4} = 0{.}75$$

Claim 4 is an extrapolation that mixes information from C5 imprecisely, a partial hallucination.

Context Precision:

Relevant chunks: C1, C2, C4 (3 out of 5).

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

C3 and C5 are noise. C5 in particular contributed to the partial hallucination about JSON.

Context Recall:

Ground-truth claims:

“Redis operates in memory, GET < 1 ms” → attributable to C1 ✓
“P99 < 1 ms on LAN with 10 GB” → attributable to C4 ✓
“Clustering scales beyond the RAM of a single node” → attributable to C2 ✓

$$\text{ContextRecall} = \frac{3}{3} = 1{.}00$$

Retrieval found every chunk needed for the ground truth. The problem is not recall but precision (C3 and C5 contaminated the context).

Answer Relevance:

The judge generates 3 hypothetical questions from the answer:

“What latencies does Redis offer for in-memory reads?” — cos = 0.91
“How does Redis scale horizontally?” — cos = 0.74
“Which JSON modules does Redis include?” — cos = 0.52 (a drift caused by the hallucination)

$$\text{AnswerRelevance} = \frac{0{.}91 + 0{.}74 + 0{.}52}{3} = 0{.}72$$

The drift towards JSON reduced relevance. A tighter answer would have scored around 0.90.

Summary of the example

Metric	Value	Diagnosis
Faithfulness	0.75	LLM extrapolated beyond the context
Context Precision	0.60	Retrieval returned 2 irrelevant chunks
Context Recall	1.00	Retrieval captured everything needed
Answer Relevance	0.72	Answer drifts off topic

Main corrective action: improve the reranker so that C3 and C5 are filtered before they reach the LLM. The faithfulness and relevance problem is a direct consequence of the low precision, not of the LLM itself.

Building the golden dataset

What it is and why it matters

The golden dataset is a set of (question, relevant chunks, correct answer) tuples that acts as a reference calibrator. Without it, context recall cannot be computed (there is no ground truth) and the other metrics lack any interpretive anchor: is 0.75 faithfulness good or bad for this corpus and this domain?

A well-built golden dataset lets you:

Compare pipeline versions (embedder v1 vs v2, chunk size 512 vs 1024)
Detect regressions in CI before deploying
Stratify the analysis by question type

LLM-assisted construction pipeline

Purely manual construction is expensive. The standard pattern in 2026 is LLM assistance with human review of a sample:

Step 1 — Seed chunk selection. From the full corpus, select representative chunks through stratified sampling (by section, date, document type). For a technical corpus of 10,000 chunks, 500-1,000 seeds is a reasonable starting point.

Step 2 — Question generation. A powerful LLM (Llama-3.1-70B or similar) generates 2-3 questions per seed chunk using a prompt along these lines:

Given the following documentation fragment, generate specific questions
that can only be answered correctly using THIS fragment and not
general knowledge. The questions should be the ones an engineer looking
for operational information would ask.
Fragment: {chunk}

Step 3 — Reference answer generation. The same LLM, with access to the seed chunk (and to adjacent chunks if the question requires them), generates the reference answer.

Step 4 — Human review of a sample. Manually review 10-20 % of the generated dataset. The most common rejection criteria: trivial questions that any LLM answers without the corpus, answers the LLM padded with parametric knowledge instead of the chunks, and badly formulated or ambiguous questions.

Minimum size

Use case	Minimum pairs	Notes
Prototype / initial validation	50-100	Enough to spot coarse problems
Technical corpus in production	200-500	Allows basic stratification
Robust production with full stratification	500-1,000+	Needed to detect subtle regressions

Stratifying the dataset

A flat golden dataset measures the average but hides the extreme cases. The minimum recommended stratification includes three question types:

Easy (single-hop): A single chunk contains all the information needed. The baseline that any decent RAG must beat.
Hard (multi-hop): The correct answer requires combining information from 2-4 different chunks. This is where the limits of the reranker and the synthesis prompt show up.
Adversarial: The question has a false premise, or the corpus does not contain the answer. A correct RAG must answer “I do not have enough information”, whereas a fragile RAG hallucinates confidently. This question type directly measures the risk of high-impact hallucination.

The Goodhart trap

“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart

If you optimise the embedder or the reranker using the golden dataset as a loss function, the dataset is corrupted as a metric: the system learns to perform well on those specific questions without improving on the general domain.

The solution is the same as in supervised ML: separate the dev set (for optimisation and iteration) from the test set (for final evaluation, frozen and audited). The test set must never be used to make design decisions; only to report the state of the system at release time.

Correlation with real satisfaction

Field studies published by the Databricks teams (2024) and the RAGAS adoption analyses (2025) point to interpretable operational thresholds:

Metric range	Observable symptom	Corrective action
Faithfulness < 0.75	Users frequently report “made-up answers”	Review the LLM prompt; increase citation instructions; lower the temperature
Faithfulness 0.75-0.85	Occasional hallucinations on peripheral topics	Improve context precision to eliminate contaminating chunks
Faithfulness ≥ 0.85	Correlates with positive NPS in field studies	Hold steady; monitor for drift
Context Precision < 0.60	LLM includes contradictory information; inconsistent answers	Tune the reranker; reduce K; review similarity thresholds
Context Recall < 0.70	Multi-hop questions fail; key information missing	Review the chunking strategy; add larger chunks; enrich metadata
Answer Relevance < 0.70	Answers that are “correct but do not answer”	Review the synthesis prompt; add an explicit instruction to stick to the question

Low context precision is especially pernicious: irrelevant chunks are not neutral. They raise the probability that the LLM uses incorrect information as though it were relevant, degrading faithfulness in a chain reaction. It is the transmission belt by which a retrieval problem turns into an LLM problem.

Diagram: the continuous evaluation loop

The continuous evaluation loop: corpus → retrieval → LLM → RAGAS judge → metrics → alert → corrective action → corpus.

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

The ragas library supports asynchronous evaluation and multiple LLM backends. Integrating vLLM as the judge removes any need to send data to external APIs, which is critical in environments with sensitive data.

from ragas import evaluate
from ragas.metrics import (
 faithfulness,
 answer_relevancy,
 context_precision,
 context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Judge LLM pointing at on-premise vLLM
judge_llm = ChatOpenAI(
 model="meta-llama/Llama-3.1-70B-Instruct",
 base_url="http://vllm-service:8000/v1",
 api_key="sk-local", # vLLM ignores the value but requires the field
)

embeddings = OpenAIEmbeddings(
 model="BAAI/bge-m3",
 base_url="http://embedding-service:8001/v1",
 api_key="sk-local",
)

result = evaluate(
 dataset=golden_dataset, # HuggingFace Dataset with standard columns
 metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
 llm=judge_llm,
 embeddings=embeddings,
)

The dataset RAGAS expects has four columns: question, answer, contexts (a list of strings), ground_truth.

Langfuse for eval traceability

Each RAGAS evaluation is recorded in Langfuse as a dataset experiment, linking the scores to the production spans (see https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/). This makes it possible to correlate a drop in faithfulness with the specific request that caused it. Without that link, the metrics are numbers with no actionable context.

from langfuse import Langfuse

lf = Langfuse()

# Create or retrieve the dataset in Langfuse
dataset = lf.get_or_create_dataset("rag-golden-v3")

# Record the experiment scores
for idx, row in result.to_pandas().iterrows():
 lf.score(
 name="ragas-faithfulness",
 value=row["faithfulness"],
 trace_id=row["trace_id"], # linked to the production span
 )

Prometheus + Grafana for operational alerts

RAGAS metrics are exposed as Prometheus gauges. A Grafana dashboard with thresholds configures alerts when faithfulness falls sustainedly below 0.80:

# Prometheus alert rule
- alert: RAGFaithfulnessLow
 expr: avg_over_time(rag_faithfulness_score[30m]) < 0.80
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "RAG faithfulness below threshold ({{ $value | humanize }})"
 description: "Review context precision and the reranker. Possible corpus drift."

Running RAGAS against on-premise vLLM — practical considerations

Judge size: Llama-3.1-70B as the judge produces results comparable to GPT-4 on faithfulness and context evaluation, according to the RAGAS 0.2 benchmarks (2025). Smaller models (8B-13B) degrade judge quality on multi-hop questions.
Throughput: On on-premise hardware with 4×H100 SXM (320 GB, NVLink), a run of 200 evaluations with Llama-3.1-70B takes roughly 8-12 minutes with batch_size=8 and vLLM in continuous batching mode.
Cost per evaluation: With no external API, the marginal cost is electricity plus GPU amortisation. With 4×H100 at around 3 kW sustained, a run of 200 evaluations costs < 0.10 € in energy at a typical industrial tariff.
Recommended frequency: weekly offline eval over the full golden dataset plus sampled online eval (5-10 % of production requests) with a subset of metrics that do not require ground truth (faithfulness, answer relevance).

What we have not covered

Alternatives to RAGAS: TruLens (evaluation with modular feedbacks), DeepEval (programmatic assertions, pytest integration), ARES (Stanford framework with trained classifiers instead of LLM-as-judge), and the OpenAI evals framework. Each has different trade-offs in judge cost, reliability and ease of integration.
Continuous eval in production: automatically sampling real requests, anonymising them, running a subset of metrics without ground truth and using the result to detect system drift before users report it. It requires a data pipeline separate from the inference pipeline.
Multilingual eval: RAGAS with a judge in Spanish or Catalan over a non-English corpus has documented biases when the judge is a model trained fundamentally in English. The semantic similarity embeddings for answer relevance are especially sensitive to corpus language vs judge language.
A/B testing of RAG configurations: using RAGAS metrics as the success criterion in controlled experiments, chunk size 512 vs 1024, pure BM25 vs hybrid, cross-encoder reranker vs biencoder, with statistical significance computed over the golden dataset.

References

Es Shahul, et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 (2023). https://arxiv.org/abs/2309.15217
RAGAS Documentation v0.2. Metrics Reference. https://docs.ragas.io/en/stable/concepts/metrics/ (accessed June 2026)
Langfuse. Dataset Experiments. https://langfuse.com/docs/datasets/overview (accessed June 2026)
Databricks. LLM Quality Evaluation: From Lab to Production. Databricks Engineering Blog (2024).
Saad-Falcon, J. et al. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476 (2023).
Goodhart, C.A.E. Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics. Reserve Bank of Australia (1975). Modern formulation of the law that bears his name.
vLLM Project. OpenAI-Compatible Server. https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html (accessed June 2026)

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

Mon, 25 May 2026 07:30:00 +0200

This is stage 3 of the LLMOps pipeline. If you arrive without context on the full journey, the six-stage LLMOps pipeline describes where Eval sits between Tune and Deploy, and the anatomy of a production LLM request shows a real eval blocking the promotion of an adapter.

TL;DR

Tracing is not evaluation. Tracing tells you what happened; eval tells you whether what happened is any good. The two layers live in tools that overlap visually (Langfuse does both), but the discipline is different: tracing is continuous capture over real traffic; eval is controlled execution against a stable dataset, with metrics that have to fail CI if they drop below a threshold. Without eval, the Tune → Deploy loop closes blind: adapter v8 goes to production because the engineer “saw it answering well” on five examples. With eval done properly, v8 only gets in if it beats v7 across a battery of 500 curated cases, evaluated by a mix of heuristics, embeddings, an LLM judge calibrated against humans, and a sample of real traffic with human annotation. This post takes apart the mechanism, the mathematics that keep you honest, the real tools in 2026 and the pitfalls that turn it into theatre.

The analogy: the academic examination board

A doctoral candidate defends a thesis. They do not defend it before a single distracted professor: they defend it before a mixed board, with a syllabus fixed in advance (not improvised on the day), and with an explicit pass mark that separates pass from fail. The board is not a single expert: it is a panel combining fast readers (the heuristics: does it have the requested format? does it include the citation?), semantic reviewers (the embeddings: does it resemble the expected answer?), an external assessor trained on the subject (the LLM judge: is it faithful to the context, is it relevant, does it sound coherent?), and human jury members on a sample of the most sensitive cases. If the candidate does not reach the mark, they are not promoted: they go back and prepare the defence again.

That analogy has three edges worth holding on to from the first minute:

The syllabus is fixed, but it is actively updated when a new topic appears in the real world that the candidate must know. If it is not updated, the exam measures something increasingly distant from what happens outside.
The board has to be trained: an LLM judge not calibrated against humans is an examiner making up the criteria.
The pass mark is published beforehand: it is not decided after seeing the result, because then it is not a pass mark, it is a justification.

These three ideas run through the rest of the post. Every tool and every piece of mathematics that follows is, at bottom, a way of operationalising them.

The mechanism itself: four layers of evaluators

An eval suite in 2026 is made up of four layers that coexist. None replaces the others; each one measures what the rest cannot measure well, and leaves unmeasured what the rest do measure well.

Layer 1 — Deterministic heuristics. Rules that return true or false with no ambiguity: the output matches a regex, contains a specific entity, does not exceed a length, follows a valid JSON schema, respects a requested format (markdown, tool_call function, mandatory citation). They are cheap, extremely fast, need neither judge nor embeddings, and catch the most frequent class of bug: the model answered in the wrong format. Their limit is obvious, in that they do not know whether the answer is correct, only whether it is well formed.

Layer 2 — Semantic metrics with embeddings. They compare the model’s output with an expected answer by computing cosine similarity between their embeddings, or by measuring whether a claim in the output is entailed by the retrieved context. They are cheap, deterministic given the embedding model, and very useful for detecting answers that drift in meaning but not in form. Their limit is equally clear: two answers can have high cosine similarity and say the opposite of each other (“the customer can cancel at any time” vs “the customer cannot cancel at any time” share 80% of their tokens).

Layer 3 — LLM-as-judge. A model, or an ensemble, evaluates the output of the model under test with a prompt designed to produce a score on a rubric. The canonical methods in 2026 are G-Eval (chain-of-thought prompting with a calibrated numeric score), Prometheus (open-source judge trained specifically for evals, reporting 0.897 correlation with humans in its v2.5 release from late 2025), and panel-of-judges setups that average votes from three heterogeneous models to reduce bias. This layer captures nuances the previous two cannot see: is it faithful to the context? is it useful? is it safe? is it complete? Its limit is cost and the need for calibration, treated as a whole section below.

Layer 4 — Humans. Trained annotators who evaluate a sample of the eval set, not all of it. They are the only layer with ultimate authority over the rubric: the LLM judge is calibrated against them, not the other way round. They are expensive (≈ 0.50–2.00 € per annotated sample when the domain is technical) and slow (a competent annotator produces 60–120 quality annotations per working day). The mistake teams make again and again is dropping this layer “because we have an LLM judge”; without humans there is no calibration, and without calibration the judge measures whatever it feels like.

Normal operation of an eval gate combines all four: heuristics remove malformed outputs before spending judge tokens, embeddings filter out the manifestly irrelevant, the judge scores the rest, and humans annotate a sample every N runs to keep the judge calibrated.

The golden dataset: a versioned syllabus

The golden dataset is the most undervalued artefact in the pipeline. It is the exam. If it is badly built, everything else, the best-calibrated judge in the world, the strictest gates, the fastest suite, measures noise. Covered at the top level in the data versioning post as one of the four artefacts to version differently; here we go into detail from the Eval perspective.

The three properties a golden dataset has to satisfy are:

Stratified representativeness. The dataset has to cover the system’s real input space in proportions that reflect production. If 30% of real traffic is in German, 30% of the golden set has to be in German; if 12% of questions are about subscription cancellation, that category cannot be 60% of the eval set just because it was easy to annotate. Stratification stays auditable: each example carries tags (lang=de, category=cancellation, tenant_type=enterprise, difficulty=hard) and the suite reports metrics per segment, not just the aggregate.

Strict holdout, no contamination with training. This rule is so obvious that almost every team believes it is following it, and almost every team breaks it without noticing. If the golden eval set gets mixed with the fine-tuning dataset, because somebody did a badly built random_split, because a purchased dataset is used by other vendors for training, because the LLM judge saw it during pretraining, then the metric stops measuring generalisation and starts measuring memorisation. The eval set hash is versioned separately (covered in the continuous fine-tuning post) and a routine leakage check is run: if an example in the golden set matches one in training token for token, it raises an alert.

Reasoned sample size, not aspirational. How many examples are needed? There is a mathematical minimum for telling two models apart with confidence. If model A is right 80% of the time and model B 85% on the same set, the 95% confidence interval for that difference of proportions (unpaired) is:

\[ \Delta p \pm 1{.}96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

To distinguish 80% from 85% with 95% confidence (an interval that does not cross zero), you need n ≈ 700 examples. To distinguish 90% from 91% under the same criterion, the calculation gives n ≈ 6,500. A “we have 50 examples in the golden set and we see adapter v8 scoring 90%” means nothing statistically: the confidence interval is ±8 points. The field’s practical rule in 2026 is a minimum of 300 examples to detect coarse differences, ideally 500–1,500 if you want to detect fine improvements, and to start by stratifying properly before obsessing over sample size.

On top of this comes active maintenance: the golden set is enriched with production incidents (covered in detail in the retrain post), so that every real complaint that ends in a bug becomes a curated example the next deploy candidate will have to pass. The golden set is not static: it is a living record of errors the system has already made and must not make again.

LLM-as-judge: how you calibrate an examiner

Layer 3 is the one most teams misuse. The typical mistake is: “we use GPT-4 as judge because it is the most capable”. The judge is not chosen for nominal capability; it is chosen for agreement with humans on the specific rubric you are measuring. A judge with 60% agreement is useless even if it is GPT-5; a Prometheus 7B fine-tuned for your domain with 88% agreement is worth more.

The standard metric for measuring agreement between two annotators (human vs judge, or two humans with each other) is Cohen’s kappa, which corrects for the agreement expected by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]

where $p_o$ is the observed proportion of agreement and $p_e$ is the proportion expected by chance under each annotator’s marginal distributions. The accepted interpretations in the literature are:

κ < 0.40: poor agreement. The judge says whatever it likes.
κ ∈ [0.40, 0.60]: moderate. Acceptable for coarse signals (is it toxic?), dreadful for nuance (is it faithful to the context?).
κ ∈ [0.60, 0.80]: substantial. Useful in production for most metrics.
κ > 0.80: almost perfect. The judge can be treated as a substitute for the human for that specific kind of judgement.

Numerically, consider a binary rubric (faithful / not faithful) over 200 examples annotated by a human and by a judge. If the human said “faithful” in 150 cases and the judge in 140, and they agree on 175 of the 200, then $p_o = 0{.}875$; the marginals are $p_h = 0{.}75$, $p_j = 0{.}70$, and $p_e = 0{.}75 \cdot 0{.}70 + 0{.}25 \cdot 0{.}30 = 0{.}600$. Kappa comes out at 0.6875: substantial but not excellent, usable, with an eye kept on the rubric.

Calibrating the judge means an explicit process:

Build a calibration set — 100–300 examples annotated by trained humans, with written guidelines. Inter-annotator kappa between the humans is also measured; if the humans themselves cannot agree, the rubric is badly written before you even start talking about a judge.
Iterate the judge prompt until judge agreement with the humans passes the accepted threshold (typically κ ≥ 0.7 for sensitive metrics).
Pin the judge version (claude-3-5-sonnet-20251022, gpt-4o-2024-11, prometheus-2-7b@sha256:…): any change invalidates the calibration.
Recalibrate periodically — every time the judge, the judge prompt, or the rubric changes. The frequency the field recommends in 2026 is quarterly at minimum, monthly if the rubric is new.
Persist everything in lineage — a score of “faithfulness 0.87” with no traceability of which judge, which prompt, which calibration set and which human validated it, is decorative.

The prompt versioning post covers how judge prompt versioning is materialised. The practical consequence is that the judge is versioned like any other model: your eval suite has judge_id = prometheus-2.5@v3 just as it has adapter_id = customer_support_v7.

The two cadences: CI gate and platform regression

Eval suites live in two places and run on two different cadences. Teams that conflate them turn one of the two into theatre.

CI gate (pre-merge, blocking). It runs on every pull request that modifies prompts, adapters, RAG configuration, or any artefact that could move the model’s output. It runs against the golden dataset versioned at the hash that is on main. The gate fails the merge if:

the critical metric drops more than X percentage points in absolute terms against the baseline (typically X = 2);
some safety metric (toxicity, PII leakage) crosses a hard threshold (typically tox > 0.02);
some strategic segment (a language, an enterprise-type tenant) drops more than Y points even if the aggregate improves.

This cadence has to be fast (ideally < 10 minutes over 500 examples) and cheap (LLM judge in batch mode, cached embeddings, heuristics run locally). The CI gate is not exhaustive: it is the low-latency line of defence.

Platform regression (post-deploy, continuous). It runs on a schedule (typically nightly or weekly) over sampled production traffic, not over the static golden set. It detects drift: the model has not changed, the golden set has not changed, but the users have, and quality over real traffic drops. This cadence is more expensive (judge over thousands of samples, human annotation over hundreds), tolerates latencies of hours, and its main consumer is not CI but the observability dashboard and the humans on the product team who decide whether to open a retrain cycle.

Both cadences persist results in the same store (Langfuse, MLflow, or equivalent) and connect them by model_id, prompt_id, dataset_hash and judge_id. Without that glue of identifiers, the metric that passes CI cannot be correlated with the one that fails in production three weeks later.

The minimum mathematics that matter

Beyond kappa and the interval for the difference of proportions, both already covered above, there are three more mathematical pieces that any team running evals seriously ends up using.

Confidence interval for a continuous metric. If your metric is a continuous score (faithfulness ∈ [0, 1]) and you measure the sample mean $\bar{x}$ over n examples with standard deviation s, the 95% confidence interval for the population mean is:

\[ \bar{x} \pm 1{.}96 \cdot \frac{s}{\sqrt{n}} \]

For n = 300 and s ≈ 0.2 (typical of a 0-1 score with non-degenerate variance), the margin is ±0.023. This means that differences below 2 hundredths cannot be distinguished from noise at that sample size. If your team is chasing “+0.5 pp” improvements over 100 examples, it is optimising noise.

Judge cost as a function of sample size and rubric. The cost of one eval pass with LLM-as-judge over n examples, with m metrics evaluated in a single call per example, and per-token price $c_{in}, c_{out}$ on the judge model, is:

\[ C \approx n \cdot (t_{in} \cdot c_{in} + t_{out} \cdot c_{out}) \]

where $t_{in}$ is the number of input tokens (includes context, output of the model under test, full rubric) and $t_{out}$ the output tokens (includes the judge’s CoT + score). For n = 500, $t_{in}$ ≈ 4,000, $t_{out}$ ≈ 300, a GPT-4o judge at May 2026 prices, one pass costs on the order of 8–15 USD per suite. If the pass fires on every PR and there are 30 PRs/day, that is 240–450 USD/day on CI gates alone. Multiplied by continuous regression, teams that do not control this spend four figures a month on judge tokens without realising. The canonical mitigation is a mix of layers: heuristics and embeddings filter first, the judge is only invoked on what the cheap layers cannot resolve, and for platform regression a self-hosted open-source judge is used (Prometheus 7B on your own GPU plane) instead of a commercial model.

Distinction between aggregate metric and per-segment metric. The classic eval fallacy is the hidden mean. If your suite reports faithfulness = 0.87 and the team reads it as “up 2 points against the previous adapter”, this may be what is happening: the new adapter gains 4 points in English (where 70% of the eval set is) and loses 6 points in German (where 30% is). The aggregate mean improves, the German experience gets worse. Any serious suite reports a breakdown by strategic segment (language, tenant type, question category, context length). The CI gate can also have per-segment thresholds, not just aggregate ones.

The 2026 stack: dominant tools

Tool	Main layer	Licence	Maintainer	When to pick it
DeepEval	CI gate	Apache 2.0	Confident AI	“Evals as pytest” — assertions in Python code, trivial integration with GitHub Actions. A reasonable default.
Promptfoo	CI gate	MIT	Promptfoo Inc.	Declarative YAML, matrix of prompts × providers × assertions, diff vs baseline. DevOps-friendly.
RAGAS	RAG-specific metrics	Apache 2.0	Exploding Gradients	Faithfulness, context relevancy, answer relevancy. The canonical piece if your system is RAG.
Inspect AI	Safety/capability evals	MIT	UK AI Safety Institute	Suite focused on safety and capability. Useful for regulatory gates under the EU AI Act.
Langfuse Evals	Platform regression	MIT (OSS) / EE	Langfuse GmbH	Integrated with tracing — datasets, runs and scores in the same UI as production traces.
MLflow GenAI Evals	Registry + evals	Apache 2.0	Databricks/LF AI	Good when you already have MLflow for classic models; “GenAI dashboard” since 3.10.
Phoenix (Arize)	Eval + visual drift	Elastic License 2.0	Arize AI	Focused on visual debugging of embeddings and drift; a complement to Langfuse, not a replacement.
Prometheus 2.5	Self-hosted OSS judge	Apache 2.0	KAIST + LG AI	Fine-tuned judge, high correlation with GPT-4 at zero cost per token when self-hosted.
G-Eval / JudgeLM	Prompting methods	— (techniques)	academic	Prompting frameworks for LLM-as-judge — they apply on top of any judge model.

The OSS catalogue by stage goes into detail entry by entry; the parallel OSS vs hyperscalers catalogue compares against Bedrock Evaluations, Vertex AI Eval Service and Azure AI Evaluation.

The canonical pattern in 2026 is hybrid: DeepEval or Promptfoo for CI gates + Langfuse Evals for platform regression + Prometheus 2.5 as a self-hosted judge + human annotation on Argilla or Label Studio for the calibration set. Swapping any of these pillars for equivalents (W&B Weave instead of Langfuse, Inspect instead of DeepEval) is style, not functionality; what matters is that the four functions are present and connected.

Applied to typical on-premise hardware

For an on-premise deployment that wants to avoid sending sensitive outputs to a commercial LLM judge (for data sovereignty, ENS, NIS2 or equivalents), the judge is hosted on your own GPU plane. Reference figures for May 2026, based on Prometheus 2.5 (Llama-3.1-8B fine-tuned as a judge) served on vLLM:

RTX 4090 (24 GB, Ada Lovelace): viable for small suites (< 200 examples) and for running the judge offline during development. Latency per evaluation ≈ 1.5–3 s with BF16; aggregate throughput on the order of 25–40 evaluations/min with batching. Useful for a developer’s local CI gates, not for platform regression.
Generic 4×H100 SXM configuration (320 GB total, NVLink): runs the judge in parallel at TP=2 across two GPUs, leaving two free to serve the model under test. Aggregate throughput on the order of 200–350 evaluations/min, a full 500-example suite in 2–3 min. This allows PR gates with no perceptible waiting and nightly platform regression over thousands of samples with no per-token cost.

The comparative cost arithmetic is stubborn: self-hosting Prometheus 2.5 pays off an H100 in roughly 6 months if the team fires ≥ 30 PRs/day with eval gates. Below that volume, the commercial judge still wins unless sovereignty is a requirement, and under ENS / NIS2 it is.

The seven pitfalls that kill this stage

Pitfall 1 — Aged golden dataset. It is not enriched with production incidents. After months it measures a world that no longer exists. The metric climbs comfortably while real users complain more.

Pitfall 2 — Contaminated or uncalibrated judge. The LLM judge evaluates with criteria it invents itself. Without a human calibration set as reference, there is no way to know whether its 0.89 is generous, severe or random.

Pitfall 3 — Insufficient sample size. A 50-example suite, 1-point differences the team treats as significant. The confidence interval is ±10 points. They are measuring noise and making real decisions on it.

Pitfall 4 — Runaway cost. A GPT-4 batch-mode judge fired on every PR over 1,000 examples, with no prior filtering by cheap layers. The eval bill overtakes production serving. It happens more often than people admit.

Pitfall 5 — Aggregate metric hiding segments. Global mean improves 2 points, German drops 6, enterprise tenants drop 3. With no explicit per-segment breakdown, the gate approves what it should not.

Pitfall 6 — Judge with a floating version. Judge model updated without recalibrating the rubric. Thresholds lose their statistical meaning. Last month’s regressions are not comparable with those from two months ago.

Pitfall 7 — Eval gate that is not enforced. The gate exists in the documentation but not in the real workflow: the suite takes 30 minutes, developers skip it with --no-verify, managers ask for one-off exceptions that become the norm. An eval gate without enforcement is ornament.

All seven are operational, not technical. The Eval layer does not break because the mathematics are wrong: it breaks because the discipline slackens. It is the same thing that happens with unit tests in any project that grows, except that here, without the discipline, the system improves its metrics while getting worse, and that makes the degradation invisible until it is already ungovernable.

What we have not covered (upcoming posts)

Guardrails and online safety: the inline eval layer that filters outputs in real time, not in CI. Conceptually a cousin of Eval, but with very different latency constraints. Covered in detail in the guardrails post: four lines of defence (input, retrieval, tool, output), OWASP LLM Top 10 (2025), OSS catalogue and deployment patterns.
Judge ensembles and agreement between multiple judges: how to reduce bias by combining three heterogeneous models as a panel, which aggregation function works (simple majority, trimmed mean, calibrated meta-judge).
Meta-eval: how you evaluate the suite itself. If the eval improvement from adapter v7 to v8 does not translate into a real improvement for the user, the suite is wrong, and that is also measured, by correlating eval metrics against product metrics.
Metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, alternative language) as an extra gate. Beyond nominal agreement, it measures consistency.
Adversarial evals with red teaming: introducing prompt injection and jailbreak attacks as part of the gate.
Privacy in the judge: how to stop sensitive outputs from the model under test travelling to an external judge when regulation forbids it — homomorphic judge, judge in a TEE, or simply a self-hosted judge.

References

G-Eval: Liu et al., “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” — the reference paper for the chain-of-thought LLM judge method.
Prometheus: Kim et al., “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models” — OSS judge with a reported 0.897 correlation vs humans.
RAGAS: Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation” — the paper that standardises faithfulness, context relevancy and answer relevancy.
Cohen’s kappa: Cohen, J. (1960). “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement — the classic metric for inter-annotator agreement, still the operational reference.
DeepEval docs: https://docs.confident-ai.com/
Promptfoo docs: https://promptfoo.dev/docs/
Langfuse Evals: https://langfuse.com/docs/scores
Inspect AI: https://inspect.ai-safety-institute.org.uk/
EU AI Act, relevant articles on mandatory evaluation of high-risk systems — pending publication of the CEN/CENELEC technical standards on conformity assessment for GenAI.

Data versioning for LLMOps: DVC, lakeFS and the reproducible golden dataset challenge

Fri, 22 May 2026 11:00:00 +0200

TL;DR

The Data stage of the six-stage LLMOps pipeline has a silent link on which everything else depends: versioning the datasets with the same discipline the code is versioned with. It is not optional. An LLM system in production consumes at least four distinct types of dataset (training/fine-tuning, RAG corpus, golden eval set, enriched dataset from the Retrain loop) and each has its own demands. Git solves code but fails on data for two technical reasons (size and a useless binary diff) and one operational one (it does not propagate lineage as far as the weights bucket of the trained model). The two dominant OSS tools, DVC and lakeFS, unified in November 2025 under a single organisation with a roadmap oriented towards LLM training and RAG datalakes; they are still complementary projects (file-level vs whole-bucket branching) but now under common governance. The productive pattern the market has consolidated on: identify each artefact with an immutable (dataset_id, version), propagate the pair as far as the experiment tracking (MLflow / W&B), version the dataset’s schema too (not just its content), apply a strict holdout to the golden eval set so you are not measuring memorisation, and maintain bidirectional traceability dataset_version ↔ model_version ↔ deployment ↔ trace_id. Without this, the promise of “we can audit which model answered what” collapses at the first serious incident.

You are here: Data (with a cross-cutting effect on Tune, Eval and Retrain)

This post goes into the detail of the versioning link inside stage 1 · Data. Versioning belongs operationally to Data, but the artefacts it produces travel to Tune (training set), Eval (golden set) and Retrain (enriched dataset). That is why the diagram marks Data as active and shows a cross-cutting band indicating end-to-end lineage.

The master analogy: batch traceability in a serious factory

A serious pharmaceutical factory does not produce without batch traceability. Every box of pills carries a printed batch number; that batch is associated with manufacturing dates, with the specific batches of each raw material used, with the quality tests it passed, and with the technicians who signed off each step. If a patient reports an adverse effect, the factory can rewind within hours: this package → this batch → these raw materials → this shift → this production line → this quality control result. Without that chain, the incident is a permanent mystery.

A serious LLM system works the same way. The “package” is the answer a user saw in production. The “batch” is the combination of model, adapter, prompt, context and configuration that generated it. And the “raw materials” are the datasets: the training set the base model was trained on, the adapter’s fine-tuning dataset, the RAG corpus that feeds retrieval, the golden eval set that validates promotion. If a customer says “what data was the model trained on that answered X to my question Y on 14 March?”, without batch traceability the answer is “we do not know”. And that, with a customer under compliance pressure, kills the contract.

Git versions the recipe (the code). Data versioning versions the ingredients. Without both, there is no auditable factory.

The four artefacts worth versioning (with differentiated demands)

Not all datasets are versioned the same way or at the same frequency. The typical LLM system in production handles four artefacts worth governing separately.

Artefact	What it is	Typical size	New-version frequency	Who consumes it
Training / fine-tuning dataset	Input/output pairs (or conversations) that train the adapter or the model.	10⁴ – 10⁷ examples · 1 – 100 GB	Per Tune experiment	Trainer (Axolotl, TRL, Unsloth)
RAG corpus	Indexed documents that feed retrieval.	10⁵ – 10⁹ chunks · 10 GB – 10 TB	Almost continuous (streaming ingest)	Indexer + vector store
Golden eval set	Curated examples with an expected answer to measure quality.	10² – 10⁴ examples · MB	Per product release	Eval gates in CI
Enriched retrain dataset	Cases where the system failed + the human correction.	Hundreds to thousands per quarter	Per retrain cycle	Next Tune

All four have common requirements (immutable identity, lineage, schema) and relevant differences:

The training set tends to be large, stable per experiment, and the cost of an error is a lost experiment (expensive but bounded).
The RAG corpus is enormous, in continuous change, and its versioning is handled through periodic snapshots of the index (not of the raw text). Usually lakeFS or bucket branches; DVC is not the best fit.
The golden eval set is small but critical: errors here contaminate the whole promotion chain. Here the rigidity of the versioning matters more than anywhere else.
The enriched retrain dataset is incremental by nature: each Retrain cycle contributes a delta over the previous one. The new version does not overwrite; it inherits and adds.

Confusing them, treating the RAG corpus as if it were the training set, or the golden eval as if it were just another dataset, is the source of half the operational problems in data versioning.

Why Git is not enough

The obvious question: if Git already solves code, why does it not solve data too? Three reasons, two technical and one operational.

Reason 1: size. A Git repository with a 50 GB dataset becomes unmanageable. git clone pulls the entire history; git status walks every file; the pack file in .git/objects bloats to up to twice the size of the dataset. Git LFS solves the first part (the binary leaves the pack) but introduces its own complexity without addressing the other two reasons.

Reason 2: useless binary diff. Git assumes text diffs are useful. When a column changes in an 8 GB parquet, the diff is opaque, because the file is binary, compressed and columnar. You cannot do code review on a dataset change the way you do on a function change. You need a semantic diff: how many rows changed, which columns changed, which distribution moved. No native Git gives you that.

Reason 3: lineage that crosses repository boundaries. This is the most important and the most subtle. The training dataset lives in a bucket. The trainer code lives in a Git repo. The trained model is published to a model registry. Inference in production generates traces in an observability system. Connecting dataset_v3 → adapter_v7 → deployment_d2 → trace t_x9 requires propagating identifiers across four different systems, not inside one repo. Git has no opinion on this.

Data versioning tools (DVC, lakeFS, Pachyderm, Quilt) exist because they solve the three problems at once: they hang the data outside the Git repo, they offer some form of semantic diff, and they expose stable identities that can be propagated towards experiment tracking and the model registry.

DVC vs lakeFS before the unification

Until November 2025, the two dominant OSS tools coexisted as complementary approaches.

Axis	DVC	lakeFS
Mental model	“Git for data”	“Branching for the data lake”
Granularity	Individual file	Whole bucket (with per-branch namespacing)
Storage	Remote-agnostic (S3, GCS, Azure, MinIO, SSH)	S3-compatible (S3, MinIO, Ceph)
Workflow	`dvc add` + `dvc push` + `dvc.yaml` pipelines	`lakectl commit` + branches/merges over the bucket
Diff	File hash + external metadata	Object-level diff + commit log
Strong cases	Discrete training datasets, model files, reproducible pipelines	Large RAG corpora, branching a shared data lake, parallel experiments without duplicating data
Git integration	Deep (the `.dvc` files are committed to Git)	Tangential (lakeFS lives alongside)
Who operates it	MLE team	Data engineering team

In practice, many teams used them at the same time: DVC for the discrete datasets that fed an experiment (they fit in a Git repo thanks to the indirection of the .dvc pointers), and lakeFS for the large RAG corpus bucket they wanted to branch without duplicating terabytes.

What changed with the November 2025 acquisition

lakeFS acquired DVC in November 2025. The operational consequence as of May 2026 is modest but relevant:

There is (as yet) no technical merger of the projects. DVC is still DVC and lakeFS is still lakeFS. The current CLIs, formats and workflows have not changed.
An explicit combined roadmap towards LLM training and RAG datalakes. The merged organisation has stated specific priorities: consistent branching between the dataset and the trained model, native integrations with MLflow / W&B / Langfuse, support for the typical LLM formats (jsonl, parquet with embedded tokenisation), and branch-aware vector indexing.
Convergence expected in 2026-2027. The market anticipates a single registry with two operating modes (file-level + bucket-branching) under a unified CLI. As of today, teams still combine both.

The practical reading for 2026: adopt DVC for discrete training/eval datasets and lakeFS for the RAG corpus, but design the lineage so that a future unified registry can absorb both without re-versioning everything. Concretely: use stable identifiers (dataset_id, version, commit_hash) that can be propagated regardless of the tool.

The operational pattern: four-hop lineage

Once you accept that datasets have to be versioned, the question is not “which tool” but “which chain of identifiers connects production to the source data”. The pattern the market has consolidated on has four hops:

(dataset_id, dataset_version)
│ versioned in DVC or lakeFS
▼
(model_id, model_version)
│ registered in MLflow / W&B with the dataset as input
▼
(deployment_id, prompt_version)
│ registered in the model registry + prompt registry
▼
(trace_id)
│ emitted by the inference engine with OTel
▼
answer visible to the user

Each arrow is a metadata write that crosses the boundary between two systems. If a single arrow is missing, the lineage breaks and the promise of auditability evaporates.

A concrete example of the flow, using DVC + MLflow:

# Data stage: version the dataset
dvc add data/finetune_v3.jsonl
git add data/finetune_v3.jsonl.dvc data/.gitignore
git commit -m "data: finetune dataset v3"
dvc push # uploads the binary to the remote (MinIO/S3)

# Tune stage: train while recording lineage
mlflow run train.py \
 -P dataset_id=finetune \
 -P dataset_version=v3 \
 -P dataset_hash=$(dvc get-url data/finetune_v3.jsonl | sha256sum)
# The run records: input dataset + model output

# Eval stage: validate while recording lineage
mlflow run eval.py \
 -P model_id=adapter_customer_v7 \
 -P golden_set_id=customer_support \
 -P golden_set_version=v12

# Deploy stage: the deployment inherits the dataset + golden ids
# Every trace in Observe carries model_version + prompt_version
# which rewind back to dataset_version

The equivalent version with lakeFS over the RAG corpus:

# Branch for the new corpus embeddings
lakectl branch create lakefs://corpus/embed-2026q2 --source main

# Index the corpus on that branch
python index_corpus.py --branch embed-2026q2

# Validate before merging to main
python eval_retrieval.py --branch embed-2026q2 \
 --metric recall@10 --threshold 0.78

# If it passes, merge (changes the corpus that serves production)
lakectl commit lakefs://corpus/embed-2026q2 -m "embed: corpus 2026q2"
lakectl merge lakefs://corpus/embed-2026q2 lakefs://corpus/main

The virtue of the second flow: while the new corpus is being validated, the production system keeps serving from main with no interference. The parallel branch works as a real staging area over the whole bucket.

Schema contracts: data versioning without them is an illusion

Versioning a dataset’s content without versioning its schema is a frequent mistake. The problem: a versioned dataset with an implicit schema still breaks silently when a producer (the ingestion team, the annotation team, an ad-hoc script) changes a field.

A concrete case: a customer support golden eval set, 1000 examples, an expected_output field originally string. Someone decides they need to capture several valid answers and changes the field to list[string]. The eval loader accepts both formats by chance (Python is lax) but the downstream LLM judge receives a different object. The eval keeps passing but now it measures something else.

Productive pattern: the dataset is versioned with DVC/lakeFS and its schema is versioned with a Schema Registry (Confluent or Apicurio) or, in less mature systems, with a JSON Schema embedded alongside the dataset. CI blocks any PR that breaks the contract without a version bump.

A minimal schema for a golden eval entry (illustrative):

$schema: https://json-schema.org/draft/2020-12/schema
$id: https://example.org/schemas/golden_eval_entry/v3.json
type: object
required: [example_id, input, expected_outputs, rubric, segment]
properties:
 example_id: {type: string, format: uuid}
 input:
 type: object
 required: [user_query, retrieved_context]
 properties:
 user_query: {type: string}
 retrieved_context: {type: array, items: {type: string}}
 expected_outputs:
 type: array
 minItems: 1
 items: {type: string}
 rubric:
 type: object
 required: [must_include, must_not_include, format]
 properties:
 must_include: {type: array, items: {type: string}}
 must_not_include: {type: array, items: {type: string}}
 format: {enum: [text, json, markdown]}
 segment: {type: string}
 difficulty: {enum: [easy, medium, hard]}
 added_at: {type: string, format: date-time}
 curated_by: {type: string}

Operational rules:

Explicit forward/backward compatibility: adding an optional field is backward-compatible; removing a required one is breaking. The policy is enforced with a compatibility check in CI.
The schema version embedded in every row of the dataset (a _schema_version field). The loader validates that the version matches what the consuming code expects.
The schema registry as the single source of truth, not as an optional copy of the JSON Schema in four repos.

Without this level of discipline, “we have data versioning” means “we store the bytes but we do not control what they mean”.

Golden eval set: the most critical version

Of the four artefacts, the golden eval set is the one that demands the most rigour. A failure here contaminates the whole promotion chain: if the eval lies, the gates approve models that should not pass.

Three extra disciplines over the golden set:

Annotation with measured quality. Each example is labelled by a human, and a percentage (10-20 %) is annotated by two independent people. The inter-annotator agreement (Cohen’s kappa or pairwise F1) is measured and published; a golden set with kappa < 0.7 is measuring human noise, not model behaviour. Argilla and Label Studio provide the mechanics; what matters is the discipline, not the tool.

A strict holdout against contamination. The golden set must never enter the training set. The concrete mechanism: hash each golden set input (sha256 normalised by lowercasing + stripping trivial punctuation) → check in CI against all the training set hashes. If there is an intersection, CI blocks until it is resolved. Without this check, the model passes the eval by memorisation, not by capability. The consequence in production is disastrous: the “validated” model fails on cases analogous to the golden set that it had not memorised.

Additive versioning, never destructive. When the golden set grows (each retrain cycle adds cases), golden_v3 = golden_v2 ∪ new_examples. Never golden_v3 = a different new set. Only that way can you compare two models trained months apart over the same base + the new delta. If you rewrite the golden set, you cannot tell whether the March model was worse than the May one or whether you were simply measuring different things.

Summary table of the discipline per artefact:

Practice	Training set	RAG corpus	Golden eval set	Enriched retrain
Immutable versioning	Yes	Yes (snapshots)	Yes, critical	Yes
Schema with a contract	Yes	Recommended	Yes, critical	Yes
Double annotation	No	Not applicable	Yes (10-20 %)	Yes (10-20 %)
Holdout vs other datasets	N/A	N/A	Yes, hash check	Yes (vs golden)
Drift check vs previous version	Recommended	Yes	Recommended	Yes
Lineage to deployment	Yes	Yes	Yes	Yes

Promotion gates: the dataset is promoted like the model

A candidate dataset (a freshly enriched golden_v13, an enriched_retrain_2026_q2 resulting from the Retrain cycle) does not enter production just by being in the bucket. It goes through gates equivalent to the model’s or the prompt’s:

Schema validation — the contract is met. Blocked in CI if not.
Quality validation — a random 5-10 % sample reviewed by a human with a quality score ≥ 4/5. Blocked if the sample fails.
Holdout segregation check — for golden sets and enriched datasets, a hash check against all the other active datasets. Blocked if there is overlap.
Drift check vs previous version — a KS test over the distribution of the input embeddings, or simpler metrics (mean length, segment distribution, the ratio of each label). A warning if the drift is high with no documented cause; a block if it is very high.
Lineage check — the dataset explicitly declares which version it inherits from and what changed. Without that metadata, it does not go in.

Only when all five gates pass is the dataset tagged production-ready and the downstream pipelines that depend on it unblocked (the next Tune, the next product release, the next eval cycle).

The on-premise stack applied

On a generic infrastructure with an RTX 4090 (24 GB VRAM, a development / small-batch profile) and a 4×H100 SXM cluster (80 GB VRAM each, NVLink, training and production inference), data versioning fits without a dedicated GPU for the versioning itself, since versioning lives on CPU + storage, but it does touch the GPU for the drift checks that require embeddings.

Typical topology:

┌────────────────────────────────────────────────────────────┐
│ Object store (MinIO or Ceph) │
│ buckets: /training-sets /corpus-rag │
│ /golden-evals /enriched-retrain │
└────────────────────────┬───────────────────────────────────┘
│
┌─────────────────┼──────────────────┐
│ │ │
┌───▼────┐ ┌────▼────┐ ┌────▼─────┐
│ DVC │ │ lakeFS │ │ MLflow │
│ remote │ │ branches│ │ Tracking │
└───┬────┘ └────┬────┘ └────┬─────┘
│ │ │
└─────────────────┴──────────────────┘
│
┌──────▼──────┐
│ CI/CD gates │
│ (Forgejo / │
│ GitLab) │
└──────┬──────┘
│
┌──────────┴───────────┐
│ │
┌─────▼──────┐ ┌─────▼─────┐
│ RTX 4090 │ │ 4×H100 │
│ (drift │ │ (training │
│ embeds, │ │ + │
│ validates)│ │ serving) │
└────────────┘ └───────────┘

Operational notes:

The object store (MinIO or Ceph) serves both as the DVC remote and as lakeFS storage. One storage plane, two views.
The schema checks and holdout hashing are fast CPU-bound tasks; the CI runner executes them without a GPU.
The embedding drift check needs an encoder; the RTX 4090 handles this without touching the production cluster. A small encoder (BGE-small, E5-small, ~100M parameters) processes 10⁴ examples in a few minutes.
The H100 cluster stays free for training and serving, with no contamination from versioning jobs.

When do you NOT need DVC/lakeFS?

There is an opposite position defended with figures in the continuous fine-tuning post: for small systems with a single team, datasets < 1 GB and a handful of adapters, Postgres + pgvector + an S3 bucket + hashed filenames are enough. The operational complexity of DVC/lakeFS does not pay for itself.

The dividing line is reasonable:

You do not need DVC/lakeFS: a single team, small datasets, few adapters, no multiple products sharing data.
You do need it: multiple teams, datasets > 10 GB, several products sharing a golden eval set, external compliance demanding batch traceability, or an institutionalised quarterly retrain cycle.

Adopting DVC + lakeFS before you need them is overhead. Adopting them six months late is losing six months of lineage irrecoverably.

Seven pitfalls that turn data versioning into theatre

Versioning the data but not the schemas. The content is versioned, the contract changes silently, the system breaks without the versioning capturing it. A Schema Registry is not optional; it is half the problem.
The same S3 path overwritten. “Upload training.jsonl to the bucket” and the next experiment rewrites the file. S3 versioning (if it is enabled) saves your skin, but without an immutable identifier propagated to MLflow you cannot rewind. The correct pattern: training_v3.jsonl or training/2026q2/<sha>.jsonl, never the same name.
A golden eval set without a strict holdout. Without a hash check against training, the model memorises the eval and passes without having learned. It is the LLM equivalent of an exam the teacher announces in advance: everyone passes, nothing has been measured.
Not recording dataset → model lineage. When an incident requires knowing what data a certain model was trained on, the correct answer is a query to MLflow / W&B. If the answer is “let’s ask whoever trained it” (assuming they are still on the team), the lineage does not exist.
DVC added six months late. Adopting versioning in month 1 = a nuisance. Adopting it in month 6 = an irrecoverable loss of six months of datasets that can no longer be reconstructed. The curse of “we’ll add it later”.
lakeFS with branches that never get merged. Parallel branches over the corpus are useful for experimenting; kept indefinitely without merging, operations turn into a graveyard of half-updated branches. Explicit policy: merge or destroy within N weeks.
Schema validation only in production. The contract is validated when the dataset is already in production and the model already trained. By then, the incident has already happened. The validation has to be in CI, before the merge, over the delta the PR introduces.

A dataset’s cycle on one screen

┌─────────────────────────────────────────────────────────────┐
│ Producer (ingest / annotation / retrain loop) │
└────────────────┬────────────────────────────────────────────┘
│
▼ (commit a candidate version)
┌─────────────────────────┐
│ CI gates │
│ - Schema validation │
│ - Quality sampled │
│ - Holdout hash check │ ── fails → PR blocked
│ - Drift vs previous │
│ - Lineage declared │
└────────────┬────────────┘
│ passes
▼
┌─────────────────────────┐
│ DVC tag or lakeFS commit│
│ + MLflow registry │ ← immutable version
│ + Schema Registry │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Downstream pipeline │
│ Tune / Eval / Deploy │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Production trace │
│ → rewinds to dataset │
└─────────────────────────┘

What we have not covered

At top level, this post leaves out:

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal and deserves separate treatment (recall, ANN parameters, index branching vs full re-embedding).
Standardised lineage tooling (OpenLineage, Marquez): how to emit and consume lineage events interoperably between systems.
Data quality frameworks (Great Expectations, Soda, Deequ): how to write suites of “expectations” over a dataset and enforce them on every version.
Privacy-preserving versioning: federated learning without centralising the dataset, differential privacy applied to the version that gets distributed.
Contamination between third-party golden sets (HumanEval, MMLU, etc.) and the training datasets of open models: the problem of “the model passes HumanEval because HumanEval is in its pretraining”.

Each one is worth a post of its own when the field justifies it.

References

DVC documentation — versioning workflows, pipelines and remotes.
lakeFS documentation — branching, merging and commits over the bucket.
lakeFS acquires DVC, November 2025 — announcement and combined roadmap.
Confluent Schema Registry and Apicurio — schema contracts for streaming data.
OpenLineage and Marquez — open standard for lineage events.
Great Expectations — data quality expectations in CI.
MLflow Tracking — input datasets as first-class artefacts since MLflow 2.4.
Pachyderm and Quilt — historical alternatives to DVC/lakeFS.
On eval set contamination: “Stop Uploading Test Data in Plain Text” (Magar & Schwartz, 2022) and later work on detecting contamination in pretraining corpora.

Golden-Dataset on lo0 — Tech Blog

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

TL;DR

The master analogy: the quality inspector at a furniture factory

The problem of evaluating RAG

The four RAGAS metrics

1. Faithfulness — fidelity to the context

2. Answer Relevance — relevance of the answer

3. Context Precision — retrieval precision

4. Context Recall — retrieval recall

5. Noise Sensitivity — sensitivity to noise

A complete worked example

Scenario

Step-by-step calculation

Summary of the example

Building the golden dataset

What it is and why it matters

LLM-assisted construction pipeline

Minimum size

Stratifying the dataset

The Goodhart trap

Correlation with real satisfaction

Diagram: the continuous evaluation loop

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

Langfuse for eval traceability

Prometheus + Grafana for operational alerts

Running RAGAS against on-premise vLLM — practical considerations

What we have not covered

See also

References

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

TL;DR

The analogy: the academic examination board

The mechanism itself: four layers of evaluators

The golden dataset: a versioned syllabus

LLM-as-judge: how you calibrate an examiner

The two cadences: CI gate and platform regression

The minimum mathematics that matter

The 2026 stack: dominant tools

Applied to typical on-premise hardware

The seven pitfalls that kill this stage

What we have not covered (upcoming posts)

See also

References

Data versioning for LLMOps: DVC, lakeFS and the reproducible golden dataset challenge

TL;DR

You are here: Data (with a cross-cutting effect on Tune, Eval and Retrain)

The master analogy: batch traceability in a serious factory

The four artefacts worth versioning (with differentiated demands)

Why Git is not enough

DVC vs lakeFS before the unification

What changed with the November 2025 acquisition

The operational pattern: four-hop lineage

Schema contracts: data versioning without them is an illusion

Golden eval set: the most critical version

Promotion gates: the dataset is promoted like the model

The on-premise stack applied

When do you NOT need DVC/lakeFS?

Seven pitfalls that turn data versioning into theatre

A dataset’s cycle on one screen

What we have not covered

See also

References