Data versioning for LLMOps: DVC, lakeFS and the reproducible golden dataset challenge

Fri, 22 May 2026 11:00:00 +0200

TL;DR

The Data stage of the six-stage LLMOps pipeline has a silent link on which everything else depends: versioning the datasets with the same discipline the code is versioned with. It is not optional. An LLM system in production consumes at least four distinct types of dataset (training/fine-tuning, RAG corpus, golden eval set, enriched dataset from the Retrain loop) and each has its own demands. Git solves code but fails on data for two technical reasons (size and a useless binary diff) and one operational one (it does not propagate lineage as far as the weights bucket of the trained model). The two dominant OSS tools, DVC and lakeFS, unified in November 2025 under a single organisation with a roadmap oriented towards LLM training and RAG datalakes; they are still complementary projects (file-level vs whole-bucket branching) but now under common governance. The productive pattern the market has consolidated on: identify each artefact with an immutable (dataset_id, version), propagate the pair as far as the experiment tracking (MLflow / W&B), version the dataset’s schema too (not just its content), apply a strict holdout to the golden eval set so you are not measuring memorisation, and maintain bidirectional traceability dataset_version ↔ model_version ↔ deployment ↔ trace_id. Without this, the promise of “we can audit which model answered what” collapses at the first serious incident.

You are here: Data (with a cross-cutting effect on Tune, Eval and Retrain)

This post goes into the detail of the versioning link inside stage 1 · Data. Versioning belongs operationally to Data, but the artefacts it produces travel to Tune (training set), Eval (golden set) and Retrain (enriched dataset). That is why the diagram marks Data as active and shows a cross-cutting band indicating end-to-end lineage.

The master analogy: batch traceability in a serious factory

A serious pharmaceutical factory does not produce without batch traceability. Every box of pills carries a printed batch number; that batch is associated with manufacturing dates, with the specific batches of each raw material used, with the quality tests it passed, and with the technicians who signed off each step. If a patient reports an adverse effect, the factory can rewind within hours: this package → this batch → these raw materials → this shift → this production line → this quality control result. Without that chain, the incident is a permanent mystery.

A serious LLM system works the same way. The “package” is the answer a user saw in production. The “batch” is the combination of model, adapter, prompt, context and configuration that generated it. And the “raw materials” are the datasets: the training set the base model was trained on, the adapter’s fine-tuning dataset, the RAG corpus that feeds retrieval, the golden eval set that validates promotion. If a customer says “what data was the model trained on that answered X to my question Y on 14 March?”, without batch traceability the answer is “we do not know”. And that, with a customer under compliance pressure, kills the contract.

Git versions the recipe (the code). Data versioning versions the ingredients. Without both, there is no auditable factory.

The four artefacts worth versioning (with differentiated demands)

Not all datasets are versioned the same way or at the same frequency. The typical LLM system in production handles four artefacts worth governing separately.

Artefact	What it is	Typical size	New-version frequency	Who consumes it
Training / fine-tuning dataset	Input/output pairs (or conversations) that train the adapter or the model.	10⁴ – 10⁷ examples · 1 – 100 GB	Per Tune experiment	Trainer (Axolotl, TRL, Unsloth)
RAG corpus	Indexed documents that feed retrieval.	10⁵ – 10⁹ chunks · 10 GB – 10 TB	Almost continuous (streaming ingest)	Indexer + vector store
Golden eval set	Curated examples with an expected answer to measure quality.	10² – 10⁴ examples · MB	Per product release	Eval gates in CI
Enriched retrain dataset	Cases where the system failed + the human correction.	Hundreds to thousands per quarter	Per retrain cycle	Next Tune

All four have common requirements (immutable identity, lineage, schema) and relevant differences:

The training set tends to be large, stable per experiment, and the cost of an error is a lost experiment (expensive but bounded).
The RAG corpus is enormous, in continuous change, and its versioning is handled through periodic snapshots of the index (not of the raw text). Usually lakeFS or bucket branches; DVC is not the best fit.
The golden eval set is small but critical: errors here contaminate the whole promotion chain. Here the rigidity of the versioning matters more than anywhere else.
The enriched retrain dataset is incremental by nature: each Retrain cycle contributes a delta over the previous one. The new version does not overwrite; it inherits and adds.

Confusing them, treating the RAG corpus as if it were the training set, or the golden eval as if it were just another dataset, is the source of half the operational problems in data versioning.

Why Git is not enough

The obvious question: if Git already solves code, why does it not solve data too? Three reasons, two technical and one operational.

Reason 1: size. A Git repository with a 50 GB dataset becomes unmanageable. git clone pulls the entire history; git status walks every file; the pack file in .git/objects bloats to up to twice the size of the dataset. Git LFS solves the first part (the binary leaves the pack) but introduces its own complexity without addressing the other two reasons.

Reason 2: useless binary diff. Git assumes text diffs are useful. When a column changes in an 8 GB parquet, the diff is opaque, because the file is binary, compressed and columnar. You cannot do code review on a dataset change the way you do on a function change. You need a semantic diff: how many rows changed, which columns changed, which distribution moved. No native Git gives you that.

Reason 3: lineage that crosses repository boundaries. This is the most important and the most subtle. The training dataset lives in a bucket. The trainer code lives in a Git repo. The trained model is published to a model registry. Inference in production generates traces in an observability system. Connecting dataset_v3 → adapter_v7 → deployment_d2 → trace t_x9 requires propagating identifiers across four different systems, not inside one repo. Git has no opinion on this.

Data versioning tools (DVC, lakeFS, Pachyderm, Quilt) exist because they solve the three problems at once: they hang the data outside the Git repo, they offer some form of semantic diff, and they expose stable identities that can be propagated towards experiment tracking and the model registry.

DVC vs lakeFS before the unification

Until November 2025, the two dominant OSS tools coexisted as complementary approaches.

Axis	DVC	lakeFS
Mental model	“Git for data”	“Branching for the data lake”
Granularity	Individual file	Whole bucket (with per-branch namespacing)
Storage	Remote-agnostic (S3, GCS, Azure, MinIO, SSH)	S3-compatible (S3, MinIO, Ceph)
Workflow	`dvc add` + `dvc push` + `dvc.yaml` pipelines	`lakectl commit` + branches/merges over the bucket
Diff	File hash + external metadata	Object-level diff + commit log
Strong cases	Discrete training datasets, model files, reproducible pipelines	Large RAG corpora, branching a shared data lake, parallel experiments without duplicating data
Git integration	Deep (the `.dvc` files are committed to Git)	Tangential (lakeFS lives alongside)
Who operates it	MLE team	Data engineering team

In practice, many teams used them at the same time: DVC for the discrete datasets that fed an experiment (they fit in a Git repo thanks to the indirection of the .dvc pointers), and lakeFS for the large RAG corpus bucket they wanted to branch without duplicating terabytes.

What changed with the November 2025 acquisition

lakeFS acquired DVC in November 2025. The operational consequence as of May 2026 is modest but relevant:

There is (as yet) no technical merger of the projects. DVC is still DVC and lakeFS is still lakeFS. The current CLIs, formats and workflows have not changed.
An explicit combined roadmap towards LLM training and RAG datalakes. The merged organisation has stated specific priorities: consistent branching between the dataset and the trained model, native integrations with MLflow / W&B / Langfuse, support for the typical LLM formats (jsonl, parquet with embedded tokenisation), and branch-aware vector indexing.
Convergence expected in 2026-2027. The market anticipates a single registry with two operating modes (file-level + bucket-branching) under a unified CLI. As of today, teams still combine both.

The practical reading for 2026: adopt DVC for discrete training/eval datasets and lakeFS for the RAG corpus, but design the lineage so that a future unified registry can absorb both without re-versioning everything. Concretely: use stable identifiers (dataset_id, version, commit_hash) that can be propagated regardless of the tool.

The operational pattern: four-hop lineage

Once you accept that datasets have to be versioned, the question is not “which tool” but “which chain of identifiers connects production to the source data”. The pattern the market has consolidated on has four hops:

(dataset_id, dataset_version)
│ versioned in DVC or lakeFS
▼
(model_id, model_version)
│ registered in MLflow / W&B with the dataset as input
▼
(deployment_id, prompt_version)
│ registered in the model registry + prompt registry
▼
(trace_id)
│ emitted by the inference engine with OTel
▼
answer visible to the user

Each arrow is a metadata write that crosses the boundary between two systems. If a single arrow is missing, the lineage breaks and the promise of auditability evaporates.

A concrete example of the flow, using DVC + MLflow:

# Data stage: version the dataset
dvc add data/finetune_v3.jsonl
git add data/finetune_v3.jsonl.dvc data/.gitignore
git commit -m "data: finetune dataset v3"
dvc push # uploads the binary to the remote (MinIO/S3)

# Tune stage: train while recording lineage
mlflow run train.py \
 -P dataset_id=finetune \
 -P dataset_version=v3 \
 -P dataset_hash=$(dvc get-url data/finetune_v3.jsonl | sha256sum)
# The run records: input dataset + model output

# Eval stage: validate while recording lineage
mlflow run eval.py \
 -P model_id=adapter_customer_v7 \
 -P golden_set_id=customer_support \
 -P golden_set_version=v12

# Deploy stage: the deployment inherits the dataset + golden ids
# Every trace in Observe carries model_version + prompt_version
# which rewind back to dataset_version

The equivalent version with lakeFS over the RAG corpus:

# Branch for the new corpus embeddings
lakectl branch create lakefs://corpus/embed-2026q2 --source main

# Index the corpus on that branch
python index_corpus.py --branch embed-2026q2

# Validate before merging to main
python eval_retrieval.py --branch embed-2026q2 \
 --metric recall@10 --threshold 0.78

# If it passes, merge (changes the corpus that serves production)
lakectl commit lakefs://corpus/embed-2026q2 -m "embed: corpus 2026q2"
lakectl merge lakefs://corpus/embed-2026q2 lakefs://corpus/main

The virtue of the second flow: while the new corpus is being validated, the production system keeps serving from main with no interference. The parallel branch works as a real staging area over the whole bucket.

Schema contracts: data versioning without them is an illusion

Versioning a dataset’s content without versioning its schema is a frequent mistake. The problem: a versioned dataset with an implicit schema still breaks silently when a producer (the ingestion team, the annotation team, an ad-hoc script) changes a field.

A concrete case: a customer support golden eval set, 1000 examples, an expected_output field originally string. Someone decides they need to capture several valid answers and changes the field to list[string]. The eval loader accepts both formats by chance (Python is lax) but the downstream LLM judge receives a different object. The eval keeps passing but now it measures something else.

Productive pattern: the dataset is versioned with DVC/lakeFS and its schema is versioned with a Schema Registry (Confluent or Apicurio) or, in less mature systems, with a JSON Schema embedded alongside the dataset. CI blocks any PR that breaks the contract without a version bump.

A minimal schema for a golden eval entry (illustrative):

$schema: https://json-schema.org/draft/2020-12/schema
$id: https://example.org/schemas/golden_eval_entry/v3.json
type: object
required: [example_id, input, expected_outputs, rubric, segment]
properties:
 example_id: {type: string, format: uuid}
 input:
 type: object
 required: [user_query, retrieved_context]
 properties:
 user_query: {type: string}
 retrieved_context: {type: array, items: {type: string}}
 expected_outputs:
 type: array
 minItems: 1
 items: {type: string}
 rubric:
 type: object
 required: [must_include, must_not_include, format]
 properties:
 must_include: {type: array, items: {type: string}}
 must_not_include: {type: array, items: {type: string}}
 format: {enum: [text, json, markdown]}
 segment: {type: string}
 difficulty: {enum: [easy, medium, hard]}
 added_at: {type: string, format: date-time}
 curated_by: {type: string}

Operational rules:

Explicit forward/backward compatibility: adding an optional field is backward-compatible; removing a required one is breaking. The policy is enforced with a compatibility check in CI.
The schema version embedded in every row of the dataset (a _schema_version field). The loader validates that the version matches what the consuming code expects.
The schema registry as the single source of truth, not as an optional copy of the JSON Schema in four repos.

Without this level of discipline, “we have data versioning” means “we store the bytes but we do not control what they mean”.

Golden eval set: the most critical version

Of the four artefacts, the golden eval set is the one that demands the most rigour. A failure here contaminates the whole promotion chain: if the eval lies, the gates approve models that should not pass.

Three extra disciplines over the golden set:

Annotation with measured quality. Each example is labelled by a human, and a percentage (10-20 %) is annotated by two independent people. The inter-annotator agreement (Cohen’s kappa or pairwise F1) is measured and published; a golden set with kappa < 0.7 is measuring human noise, not model behaviour. Argilla and Label Studio provide the mechanics; what matters is the discipline, not the tool.

A strict holdout against contamination. The golden set must never enter the training set. The concrete mechanism: hash each golden set input (sha256 normalised by lowercasing + stripping trivial punctuation) → check in CI against all the training set hashes. If there is an intersection, CI blocks until it is resolved. Without this check, the model passes the eval by memorisation, not by capability. The consequence in production is disastrous: the “validated” model fails on cases analogous to the golden set that it had not memorised.

Additive versioning, never destructive. When the golden set grows (each retrain cycle adds cases), golden_v3 = golden_v2 ∪ new_examples. Never golden_v3 = a different new set. Only that way can you compare two models trained months apart over the same base + the new delta. If you rewrite the golden set, you cannot tell whether the March model was worse than the May one or whether you were simply measuring different things.

Summary table of the discipline per artefact:

Practice	Training set	RAG corpus	Golden eval set	Enriched retrain
Immutable versioning	Yes	Yes (snapshots)	Yes, critical	Yes
Schema with a contract	Yes	Recommended	Yes, critical	Yes
Double annotation	No	Not applicable	Yes (10-20 %)	Yes (10-20 %)
Holdout vs other datasets	N/A	N/A	Yes, hash check	Yes (vs golden)
Drift check vs previous version	Recommended	Yes	Recommended	Yes
Lineage to deployment	Yes	Yes	Yes	Yes

Promotion gates: the dataset is promoted like the model

A candidate dataset (a freshly enriched golden_v13, an enriched_retrain_2026_q2 resulting from the Retrain cycle) does not enter production just by being in the bucket. It goes through gates equivalent to the model’s or the prompt’s:

Schema validation — the contract is met. Blocked in CI if not.
Quality validation — a random 5-10 % sample reviewed by a human with a quality score ≥ 4/5. Blocked if the sample fails.
Holdout segregation check — for golden sets and enriched datasets, a hash check against all the other active datasets. Blocked if there is overlap.
Drift check vs previous version — a KS test over the distribution of the input embeddings, or simpler metrics (mean length, segment distribution, the ratio of each label). A warning if the drift is high with no documented cause; a block if it is very high.
Lineage check — the dataset explicitly declares which version it inherits from and what changed. Without that metadata, it does not go in.

Only when all five gates pass is the dataset tagged production-ready and the downstream pipelines that depend on it unblocked (the next Tune, the next product release, the next eval cycle).

The on-premise stack applied

On a generic infrastructure with an RTX 4090 (24 GB VRAM, a development / small-batch profile) and a 4×H100 SXM cluster (80 GB VRAM each, NVLink, training and production inference), data versioning fits without a dedicated GPU for the versioning itself, since versioning lives on CPU + storage, but it does touch the GPU for the drift checks that require embeddings.

Typical topology:

┌────────────────────────────────────────────────────────────┐
│ Object store (MinIO or Ceph) │
│ buckets: /training-sets /corpus-rag │
│ /golden-evals /enriched-retrain │
└────────────────────────┬───────────────────────────────────┘
│
┌─────────────────┼──────────────────┐
│ │ │
┌───▼────┐ ┌────▼────┐ ┌────▼─────┐
│ DVC │ │ lakeFS │ │ MLflow │
│ remote │ │ branches│ │ Tracking │
└───┬────┘ └────┬────┘ └────┬─────┘
│ │ │
└─────────────────┴──────────────────┘
│
┌──────▼──────┐
│ CI/CD gates │
│ (Forgejo / │
│ GitLab) │
└──────┬──────┘
│
┌──────────┴───────────┐
│ │
┌─────▼──────┐ ┌─────▼─────┐
│ RTX 4090 │ │ 4×H100 │
│ (drift │ │ (training │
│ embeds, │ │ + │
│ validates)│ │ serving) │
└────────────┘ └───────────┘

Operational notes:

The object store (MinIO or Ceph) serves both as the DVC remote and as lakeFS storage. One storage plane, two views.
The schema checks and holdout hashing are fast CPU-bound tasks; the CI runner executes them without a GPU.
The embedding drift check needs an encoder; the RTX 4090 handles this without touching the production cluster. A small encoder (BGE-small, E5-small, ~100M parameters) processes 10⁴ examples in a few minutes.
The H100 cluster stays free for training and serving, with no contamination from versioning jobs.

When do you NOT need DVC/lakeFS?

There is an opposite position defended with figures in the continuous fine-tuning post: for small systems with a single team, datasets < 1 GB and a handful of adapters, Postgres + pgvector + an S3 bucket + hashed filenames are enough. The operational complexity of DVC/lakeFS does not pay for itself.

The dividing line is reasonable:

You do not need DVC/lakeFS: a single team, small datasets, few adapters, no multiple products sharing data.
You do need it: multiple teams, datasets > 10 GB, several products sharing a golden eval set, external compliance demanding batch traceability, or an institutionalised quarterly retrain cycle.

Adopting DVC + lakeFS before you need them is overhead. Adopting them six months late is losing six months of lineage irrecoverably.

Seven pitfalls that turn data versioning into theatre

Versioning the data but not the schemas. The content is versioned, the contract changes silently, the system breaks without the versioning capturing it. A Schema Registry is not optional; it is half the problem.
The same S3 path overwritten. “Upload training.jsonl to the bucket” and the next experiment rewrites the file. S3 versioning (if it is enabled) saves your skin, but without an immutable identifier propagated to MLflow you cannot rewind. The correct pattern: training_v3.jsonl or training/2026q2/<sha>.jsonl, never the same name.
A golden eval set without a strict holdout. Without a hash check against training, the model memorises the eval and passes without having learned. It is the LLM equivalent of an exam the teacher announces in advance: everyone passes, nothing has been measured.
Not recording dataset → model lineage. When an incident requires knowing what data a certain model was trained on, the correct answer is a query to MLflow / W&B. If the answer is “let’s ask whoever trained it” (assuming they are still on the team), the lineage does not exist.
DVC added six months late. Adopting versioning in month 1 = a nuisance. Adopting it in month 6 = an irrecoverable loss of six months of datasets that can no longer be reconstructed. The curse of “we’ll add it later”.
lakeFS with branches that never get merged. Parallel branches over the corpus are useful for experimenting; kept indefinitely without merging, operations turn into a graveyard of half-updated branches. Explicit policy: merge or destroy within N weeks.
Schema validation only in production. The contract is validated when the dataset is already in production and the model already trained. By then, the incident has already happened. The validation has to be in CI, before the merge, over the delta the PR introduces.

A dataset’s cycle on one screen

┌─────────────────────────────────────────────────────────────┐
│ Producer (ingest / annotation / retrain loop) │
└────────────────┬────────────────────────────────────────────┘
│
▼ (commit a candidate version)
┌─────────────────────────┐
│ CI gates │
│ - Schema validation │
│ - Quality sampled │
│ - Holdout hash check │ ── fails → PR blocked
│ - Drift vs previous │
│ - Lineage declared │
└────────────┬────────────┘
│ passes
▼
┌─────────────────────────┐
│ DVC tag or lakeFS commit│
│ + MLflow registry │ ← immutable version
│ + Schema Registry │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Downstream pipeline │
│ Tune / Eval / Deploy │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Production trace │
│ → rewinds to dataset │
└─────────────────────────┘

What we have not covered

At top level, this post leaves out:

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal and deserves separate treatment (recall, ANN parameters, index branching vs full re-embedding).
Standardised lineage tooling (OpenLineage, Marquez): how to emit and consume lineage events interoperably between systems.
Data quality frameworks (Great Expectations, Soda, Deequ): how to write suites of “expectations” over a dataset and enforce them on every version.
Privacy-preserving versioning: federated learning without centralising the dataset, differential privacy applied to the version that gets distributed.
Contamination between third-party golden sets (HumanEval, MMLU, etc.) and the training datasets of open models: the problem of “the model passes HumanEval because HumanEval is in its pretraining”.

Each one is worth a post of its own when the field justifies it.

References

DVC documentation — versioning workflows, pipelines and remotes.
lakeFS documentation — branching, merging and commits over the bucket.
lakeFS acquires DVC, November 2025 — announcement and combined roadmap.
Confluent Schema Registry and Apicurio — schema contracts for streaming data.
OpenLineage and Marquez — open standard for lineage events.
Great Expectations — data quality expectations in CI.
MLflow Tracking — input datasets as first-class artefacts since MLflow 2.4.
Pachyderm and Quilt — historical alternatives to DVC/lakeFS.
On eval set contamination: “Stop Uploading Test Data in Plain Text” (Magar & Schwartz, 2022) and later work on detecting contamination in pretraining corpora.

Schema-Registry on lo0 — Tech Blog