Helm on lo0 — Tech Blog

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

Tue, 16 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a full stop. The dollar sign is not used (on this site it is a formula delimiter).

Post B7 of the benchmarking track. The B1 introduction established the distinction between the three axes (performance, quality, energy). The B2 tool catalogue covered the performance axis (TTFT, ITL, goodput). This article covers the quality axis: what each benchmark measures, how it is run, and why the leaderboard numbers do not always say what they appear to say. The LLMOps eval pipeline and LLM-as-judge cover the system’s internal evaluation layer; this article covers the evaluation of the base model with standard academic benchmarks.

TL;DR

An inference engine with an 80 ms TTFT serving a low-quality model is worth nothing: the platform’s Pareto front has two independent axes. In 2025–2026, the most widely used static benchmarks (MMLU, HumanEval) have crossed the saturation threshold (HumanEval > 93% in frontier models, MMLU-Pro > 75% in the top-5), which invalidates their ability to discriminate at the high end. The main cause is not that the models are better: it is data contamination, with 18% n-gram overlap detected in Llama-2-70B against evaluation datasets. The OSS tools active in 2026 are lm-evaluation-harness (EleutherAI, MIT, 60+ tasks, the ecosystem’s de facto engine), HELM (Stanford, Apache 2.0, in maintenance since June 2026), LiveBench (MIT, 1,000 questions refreshed monthly to avoid contamination), lighteval (Hugging Face, MIT, 1,000+ tasks, vLLM support), and LMArena/Chatbot Arena (Bradley-Terry over millions of human comparisons). OpenLLM Leaderboard v2 was archived in 2025 and replaced by the OpenEvals ecosystem (200+ specialised leaderboards). Reproducing an eval number requires pinning: harness version, model version, prompt template version and number of shots.

Quality ≠ performance: the independent Pareto axis

The performance axis (TTFT, ITL, throughput) measures how long the engine takes to produce tokens. The quality axis measures how correct those tokens are. They are orthogonal: a small model served with vLLM at 3,000 tok/s can score 42% on MMLU-Pro; a frontier model may need 4× more GPU and score 72%. The product throughput × quality is the only number that matters for a platform in production.

The operational consequence: select model and engine first by quality (does it clear the minimum threshold for the task?) and then by performance (does it meet the SLO within the GPU budget?). Inverting the order, picking the fastest engine and fitting the model into it, is the most frequent mistake in initial sizing.

Reference benchmarks: what each one measures

The table below collects the standard academic benchmarks in 2026, the capability each one measures, its test set size and the current saturation level in frontier models.

Benchmark	Capability measured	Format	Test set	Frontier saturation 2026
MMLU	Multidisciplinary knowledge (57 subjects)	4-choice MCQ	14,079	High: top-5 > 88%; insufficient separation
MMLU-Pro	MMLU with 10 options, multistep reasoning	10-choice MCQ	12,032	Medium: top-5 between 72–77%; still separates
GSM8K	Primary-school maths reasoning (word problems)	answer generation	1,319	Very high: top models > 97%
MATH	Competition maths (algebra, geometry, calculus, etc.)	LaTeX generation	5,000	Medium-high: top models 85–92%
GPQA Diamond	PhD-level scientific reasoning (biology, physics, chemistry)	4-choice MCQ	198	Emerging: top > 90% in 2026; still separates in 60–90%
BBH (BIG-Bench Hard)	23 tasks where earlier models fell below human level	generation	6,511	Medium: useful for complex reasoning
IFEval	Verifiable instruction following (format, length, language)	generation + rules	541	Low: notable separation between models
HumanEval	Python code generation (164 problems with signature)	generation + execution	164	Very high: top > 93%; saturated
MBPP	Code generation (500 problems without a function signature)	generation + execution	500	High: top models 85–92%
MuSR	Multistep reasoning over long texts	generation	~1,000	Low: useful for extended reasoning
TruthfulQA	Truthfulness: resistance to asserting popular myths	generation + MCQ	817	Medium: visible separation between families

Note on saturation: a saturated benchmark does not distinguish models at the high end; it is only valuable as a floor check (confirming the model has not regressed to the level of three years ago). For the frontier zone, the useful benchmarks in 2026 are MMLU-Pro, GPQA Diamond, IFEval and MuSR.

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

Field	Data
Repository	github.com/EleutherAI/lm-evaluation-harness
Licence	MIT
Maintainer	EleutherAI
Active version (Jun 2026)	0.4.x (`main` branch)
Available tasks	60+ benchmarks, hundreds of subtasks and variants
Supported backends	Hugging Face Transformers, vLLM, litellm, OpenAI-compatible API, GGUF
Task format	YAML + Python; each task defines dataset, template, metric and number of shots
What it measures	MCQ accuracy, exact match, pass@k on code, perplexity, BLEU/ROUGE
How it is invoked	`lm_eval --model hf --model_args pretrained=<model> --tasks mmlu,gsm8k --num_fewshot 5 --output_path results/`
Output	JSON with per-task metrics + configuration metadata
Use in the ecosystem	Official engine of the OpenLLM Leaderboard (v1 and v2); de facto reference in research

The harness is the industry standard for reproducible evaluation. Its strength is the breadth of tasks and the normalisation of the prompt format. Its limitation: the tasks are static, the datasets do not change between harness versions, which exposes it to the contamination problem if the model was trained on the same data.

Full invocation with vLLM as the backend:

lm_eval \
 --model vllm \
 --model_args pretrained=meta-llama/Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu_pro,gpqa_diamond,ifeval,humaneval \
 --num_fewshot 5 \
 --batch_size auto \
 --output_path ./eval_results/ \
 --log_samples

HELM (Stanford CRFM)

Field	Data
Repository	github.com/stanford-crfm/helm
Official site	crfm.stanford.edu/helm
Licence	Apache 2.0
Maintainer	Stanford CRFM
Status (Jun 2026)	Maintenance mode since 1 June 2026
What it measures	Accuracy, calibration, robustness, fairness, toxicity, efficiency (latency/cost)
Task format	Python + YAML; scenarios with multiple simultaneous metrics
Supported backends	API (OpenAI, Anthropic, Cohere, etc.) + local models via Hugging Face
Active variants	HELM Classic, HELM Capabilities, HELM Lite, MedHELM, HELM Safety
Installation	`pip install crfm-helm`
How it is invoked	`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000`
Output	JSON + web UI with a reproducible leaderboard and per-prompt transparency

HELM is the tool that introduced holistic evaluation (multiple metrics over the same scenario instead of a single number). In 2026, with HELM in maintenance mode, the ecosystem has fragmented into specialised leaderboards; but HELM remains the methodological reference for evaluations that need fairness and toxicity as well as accuracy.

HELM Capabilities (published March 2025): an active leaderboard with full prompt transparency, reproducible with the HELM framework. Available at crfm.stanford.edu/helm/capabilities.

LiveBench

Field	Data
Repository	github.com/LiveBench/LiveBench
Official site	livebench.ai
Licence	MIT
Maintainer	NYU, UT Austin, UC Santa Barbara
Paper	arXiv:2406.19314
Size	1,000 active questions (monthly refresh, same total size)
Categories	6: math, coding, reasoning, language, instruction following, data analysis
Tasks	18 subtasks
What it measures	General capabilities over questions with objectively verifiable answers
Anti-contamination	Questions generated from recent sources: arXiv papers, news, IMDb, new datasets
Scoring	No LLM-as-judge: automatically verifiable answer (exact ground truth)
Cadence	New questions monthly; old ones retired to avoid accumulated memorisation

LiveBench is the architectural answer to the contamination problem: if the benchmark changes every month with questions based on sources published after the cutoff of the model under evaluation, memorisation of the training set cannot inflate the result. The limitation is size (1,000 questions) and that it does not cover every domain with the same depth as MMLU.

Execution:

git clone https://github.com/LiveBench/LiveBench
cd LiveBench
pip install -e .
python livebench/gen_model_answer.py \
 --model-path meta-llama/Llama-3.1-70B-Instruct \
 --model-id llama3-70b
python livebench/gen_ground_truth_judgment.py \
 --model-id llama3-70b
python livebench/show_livebench_result.py

lighteval (Hugging Face)

Field	Data
Repository	github.com/huggingface/lighteval
Licence	MIT
Maintainer	Hugging Face (Leaderboard & Evals Team)
Available tasks	1,000+ tasks across multiple domains and languages
Supported backends	Hugging Face Accelerate (CPU/GPU/multi-GPU), Nanotron, vLLM, TGI, OpenAI-compatible endpoints
What it measures	Accuracy, BERT score, BLEU, exact match; same as the harness + HF’s own tasks
Differentiator	Designed to be the engine behind HF’s leaderboards; easy extension with custom tasks
Installation	`pip install lighteval`
How it is invoked	`lighteval accelerate –model_args “pretrained=” –tasks “lighteval
Output	JSON + direct integration with the Hugging Face Hub to publish results

lighteval is the internal successor to lm-evaluation-harness in the HF ecosystem: more modern, better vLLM and distributed backend support, and designed to run without leaving the HF stack. For teams already using the Hub to manage models, it is the lowest-friction option.

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

The OpenLLM Leaderboard v2 evaluated more than 13,000 models between June 2024 and its closure in 2025, using six fixed benchmarks: IFEval, MuSR, GPQA, MATH, BBH and MMLU-Pro, all run with lm-evaluation-harness. Its closure was a direct consequence of the saturation and gaming problem: as the benchmarks became well known, models were optimised for them.

Successor: OpenEvals (huggingface.co/spaces/OpenEvals/find-a-leaderboard)

Aspect	OpenLLM Leaderboard v2	OpenEvals
Architecture	Single centralised leaderboard	200+ specialised community leaderboards
Benchmarks	Fixed (6 tasks)	Variable per leaderboard (math, code, medical, safety, etc.)
Models evaluated	13,000+	Distributed across leaderboards
Eval engine	lm-evaluation-harness	lighteval (most) or harness
Status	Archived with an updated UI (Mar 2025)	Active; you can create your own leaderboard
Advantage	Historical consistency	Specialisation, speed of adaptation

The archived collection is available for reference at huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025.

LMArena / Chatbot Arena

Field	Data
Site	lmarena.ai (rebrand of LMSys Chatbot Arena, January 2026)
Methodology	Anonymous pairwise comparisons: the user votes for the answer they prefer without knowing the model
Statistical model	Bradley-Terry (maximum likelihood over the full history of match-ups; replaced classic Elo for greater stability)
Accumulated votes	> 6 million (Jun 2026)
Variants	Overall, Coding, Math, Hard Prompts, Multimodal — the specialised ones are more reliable
Licence/access	Free (web); vote data available under a CC licence
Limitations	Overall is biased by labs’ selective disclosure and by format gaming; Bradley-Terry was recalibrated in mid-2025 (Style Control), shifting some models by ±20–40 Elo

LMArena is the only benchmark based on human preference at scale. Its advantage: it captures real perceived quality, not just MCQ accuracy. Its limitation: it is hard to reproduce on-premise (it requires a user interface and a panel of human voters). Useful as an external validation signal, not as a CI gate.

Decision table: which tool for which objective

Objective	Tool	Why
Reproduce a standard academic eval	lm-evaluation-harness	De facto engine; maximum compatibility with papers
Holistic evaluation (accuracy + fairness + toxicity)	HELM (+ HELM Capabilities)	The only framework measuring every axis at once
Avoid contamination with recent questions	LiveBench	Monthly refresh; verifiable answers without a judge
Native integration with HF Hub and vLLM	lighteval	vLLM backend, 1,000+ tasks, direct publishing to the Hub
Compare against the public state of the art (OSS)	OpenEvals + lighteval	The successor ecosystem to the OpenLLM Leaderboard
Human preference at scale	LMArena	Bradley-Terry over millions of real comparisons
Custom domain eval (medical, legal, etc.)	lighteval with a custom task	Simpler to extend than the harness
Fast CI gate (<10 min)	lm-eval harness (subset)	Individual tasks with `--tasks ifeval,gpqa` + vLLM batching

The data contamination trap

What it is

A model that has seen a benchmark’s test data during pretraining memorises answers instead of generalising. The benchmark measures memorisation, not capability. This is called data contamination or benchmark leakage.

Scale of the problem (2024–2025 data)

Finding	Source	Figure
N-gram overlap in Llama-2-70B	“Data Contamination or Genuine Generalization?” (2025)	18.1% overlap with eval datasets
Out-of-distribution drop in contaminated large models	Ibid.	−9.4 pp on reformulated prompts vs the original test
HumanEval saturated	Artificial Analysis / multiple 2026 reports	Top models > 93%; separation ≤ 2 pp between the best 5
GPQA Diamond: appearance of indirect contamination	mindstudio.ai analysis 2026	“Data laundering”: questions leaked via online forums

Detection mechanisms

1. N-gram overlap (the classic method)

Compute the fraction of test set n-grams that appear in the model’s pretraining corpus. A practical threshold: if more than 13% of the test’s 8-grams match the corpus, the result is compromised.

\[ \text{contamination ratio} = \frac{|\{g \in \text{test n-grams}\} \cap \{g \in \text{train n-grams}\}|}{|\text{test n-grams}|} \]

Limit: it only works if you have access to the pretraining corpus. For closed models, it is not applicable.

2. Canary strings (decoy insertion)

Unique random strings are inserted into the evaluation dataset while it is being created. If the model reproduces those strings without having seen them in context, it confirms that it memorised them from pretraining. The technique was proposed in “Extracting Training Data from Large Language Models” (Carlini et al., 2021) and adopted in recent benchmarks.

3. Perturbation and reformulation

Semantically equivalent variants of the test are generated (paraphrasing, changing proper nouns, translation and back-translation). A model that generalises keeps its accuracy; a model that memorises drops. The average drop detected in contaminated models is 9–15 pp on equivalent reformulated prompts.

4. LiveBench (monthly refresh)

The architectural solution: if the questions are generated from sources published after the model’s cutoff (recent arXiv papers, news, new data), memorisation from pretraining cannot help. The model has to reason over information it could never have seen.

5. Kernel divergence analysis (Savelka et al., 2025)

A more sophisticated statistical method: it compares the embedding distribution of the test set with the distribution of the training corpus. An abnormally low divergence indicates that the benchmark and the training corpus come from the same source.

Why static leaderboards saturate

The cycle is predictable:

A new benchmark is published (MMLU in 2021, GPQA in 2023).
The labs use it as an evaluation target during pretraining and fine-tuning.
Performance on that benchmark rises quickly, faster than real capability.
The benchmark stops discriminating at the high end.
The community needs a new benchmark.

The average time between the publication of a benchmark and its saturation in frontier models has gone from ~36 months (MMLU: 2021–2024) to ~18 months (HumanEval: 2021–2023, GPQA: 2023–2025). Saturation is accelerating.

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

An eval result is only reproducible if it declares:

Parameter	Example	Why it matters
Harness version	`lm-evaluation-harness==0.4.3`	Changes to the prompt template between versions invalidate the comparison
Exact model version	`meta-llama/Llama-3.1-70B-Instruct@sha256:abc…`	Silent model updates on the Hub change the results
Prompt template	`mmlu_pro_cot_0shot` vs `mmlu_pro_5shot`	The difference between 0-shot and 5-shot can be 5–12 pp
Number of shots	`--num_fewshot 5`	The published standard varies by benchmark; 0-shot and 5-shot are not comparable

Few-shot vs zero-shot: the numerical difference

Sensitivity to the number of shots varies dramatically by benchmark:

Benchmark	Typical 0-shot	Typical 5-shot	Difference
MMLU-Pro	62%	72%	+10 pp
GPQA Diamond	55%	65%	+10 pp
GSM8K	70%	82%	+12 pp
IFEval	68%	71%	+3 pp
HumanEval	75%	80%	+5 pp

Publishing an MMLU-Pro result in 0-shot without saying so, when the leaderboard standard is 5-shot, inflates the comparison by ~10 pp. Two results from the same model published with different numbers of shots are not comparable.

Prompt sensitivity

Minor perturbations in the prompt template produce variations of up to ±8 pp on MCQ and ±15 pp on generation tasks. Sources of variance identified in the literature:

Order of the options in MCQ (position bias): up to ±4 pp.
Wording of the question (paraphrasing): ±3–8 pp.
Presence or absence of “Let’s think step by step”: ±5–12 pp on reasoning.
Language of the system instructions: ±3–10 pp in non-multilingual models.

Why two evals give different results

A common scenario: the paper reports MMLU-Pro 74.3% and the reproduction gives 71.8%. Frequent causes:

Cause	Typical magnitude
Different harness version (changed template)	1–4 pp
Model with a silent update	0.5–3 pp
Different context truncation	0.5–2 pp
Sampling seed (for generation)	0.3–1.5 pp
Tokenisation with a chat template vs without one	2–8 pp

The sum of these sources explains gaps of 2–6 pp that are not noise but systematic protocol differences.

The confidence interval formula for comparing two models

To know whether the difference between model A (accuracy $p_A$) and model B ($p_B$) over a test set of size $n$ is statistically significant:

\[ \Delta p \pm 1.96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

Applied to GPQA Diamond ($n = 198$), if $p_A = 0.90$ and $p_B = 0.88$:

\[ \Delta p = 0.02 \quad;\quad \text{margin} = 1.96 \cdot \sqrt{\frac{0.09 + 0.1056}{198}} \approx 1.96 \cdot 0.0314 \approx 0.062 \]

The 95% confidence interval is $[{-0.042},\ {+0.082}]$: it crosses zero. A difference of 2 pp over 198 questions is not statistically distinguishable from noise. To distinguish 90% from 88% with 95% confidence you need:

\[ n \approx \frac{(1.96)^2 \cdot (p_A(1-p_A) + p_B(1-p_B))}{(\Delta p)^2} \approx \frac{3.84 \cdot 0.1956}{0.0004} \approx 1{,}878 \text{ questions} \]

GPQA Diamond is not that size. Differences of 1–2 pp on GPQA Diamond are statistically invisible.

The quality axis in the scorecard

The platform’s scorecard has three columns: performance (goodput under SLO), quality (score on a reference benchmark), energy (J/token). The quality column is built like this:

Field	Content
Primary benchmark	MMLU-Pro (5-shot) — separates in the 60–77% zone
Reasoning benchmark	GPQA Diamond (0-shot) — separates in the 55–90% zone
Instruction benchmark	IFEval (0-shot prompt-level accuracy)
Code benchmark	MBPP (0-shot, pass@1) — less saturated than HumanEval
Anti-contamination	LiveBench global score (monthly)
Tool	lm-evaluation-harness (primary) + lighteval (secondary, HF Hub)
Pinned version	`harness==0.4.3`, `lighteval==0.6.x`
Shots	Declared explicitly per benchmark
Reproducibility	Output JSON stored with the model SHA
Frequency	Per model release; not in the CI of every PR (GPU-hour cost)

A full eval over 4 benchmarks (MMLU-Pro 12,032 + GPQA 198 + IFEval 541 + MBPP 500) with a 70B model on a generic 2×H100 node takes on the order of 4–8 hours depending on batch size and number of shots. The GPU cost is real: at ~5 €/h per amortised card, a full eval costs on the order of 40–80 €. It is not run on every commit; it is run on every model release or major adapter release.

Sources

EleutherAI · lm-evaluation-harness — https://github.com/EleutherAI/lm-evaluation-harness
Stanford CRFM · HELM — https://github.com/stanford-crfm/helm
Stanford CRFM · HELM Capabilities (mar-2025) — https://crfm.stanford.edu/2025/03/20/helm-capabilities.html
Stanford CRFM · HELM (sitio oficial) — https://crfm.stanford.edu/helm/
LiveBench · paper arXiv:2406.19314 — https://arxiv.org/abs/2406.19314
LiveBench · sitio oficial — https://livebench.ai
LiveBench · GitHub — https://github.com/LiveBench/LiveBench
NYU Center for Data Science · LiveBench blog — https://nyudatascience.medium.com/livebench-challenging-language-models-with-contamination-free-questions-999b52967ec8
Hugging Face · lighteval — https://github.com/huggingface/lighteval
Hugging Face · OpenLLM Leaderboard (archivado) — https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
Hugging Face · OpenEvals (sucesor) — https://huggingface.co/spaces/OpenEvals/find-a-leaderboard
Hugging Face · Archived Open LLM Leaderboard 2024-2025 — https://huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025
LMArena · Chatbot Arena — https://lmarena.ai
LMSys · blog Chatbot Arena Elo — https://www.lmsys.org/blog/2023-05-03-arena/
“A Statistical Framework for Ranking LLM-Based Chatbots” · arXiv:2412.18407 — https://arxiv.org/pdf/2412.18407
“Data Contamination or Genuine Generalization?” (2025) — https://www.suaspress.org/ojs/index.php/AJNS/article/view/v2n2a03
“When Benchmarks Leak: Inference-Time Decontamination” (2025) · arXiv:2601.19334 — https://arxiv.org/pdf/2601.19334
“How Contaminated Is Your Benchmark?” (2025) · arXiv:2502.00678 — https://arxiv.org/pdf/2502.00678
“When AI Benchmarks Plateau” (2026) · arXiv:2602.16763 — https://arxiv.org/pdf/2602.16763
GPQA · arXiv:2311.12022 — https://arxiv.org/abs/2311.12022
GPQA Diamond leaderboard · Artificial Analysis — https://artificialanalysis.ai/evaluations/gpqa-diamond
“The Emperor’s New Clothes in Benchmarking?” (2025) · arXiv:2503.16402 — https://arxiv.org/pdf/2503.16402
lm-evaluation-harness · CLI reference — https://lm-evaluation-harness.readthedocs.io/running_evals/cli_reference/
crfm-helm · PyPI — https://pypi.org/project/crfm-helm/

GitOps for the inference stack with Flux: operating the assistant as code

Thu, 11 Jun 2026 10:20:00 +0000

This post is part of the operational series on how to squeeze a generic on-premise 4×H100 SXM 80 GB LLM cluster. The sibling pieces: the document ingestion pipeline for RAG that fills the vector store we deploy here, the embeddings and reranker service with TEI that is one of the services GitOps manages, and the secrets hardening of the sovereign stack, which goes deep into the secrets problem we only state here. The complete end-to-end assistant (LibreChat + LiteLLM + RAG) that orchestrates all of this will get its own post.

TL;DR

An LLM assistant in production is a distributed system, not a binary. At a minimum it has an inference engine (vLLM or similar), an L7 gateway that routes per model and applies rate limiting, a chat front end, a vector store for RAG, an embeddings/reranker service, and an observability stack. Six substantial services, spread across three or four namespaces, with start-up dependencies (the front end is useless if the gateway is not there, the gateway is useless if the engine has not loaded the model, RAG does not answer if the vector store is empty). Operating it by hand, a kubectl apply here and a helm upgrade there, produces a cluster whose real state nobody can reconstruct: there is no record of what was applied, or when, or why. GitOps solves this with four principles (OpenGitOps): the desired state is declarative, it is versioned and immutable in git, it is applied automatically by an agent (nobody SSHes in to deploy) and it is continuously reconciled by comparing the actual state against the declared one. Flux is that agent: six controllers (source, kustomize, helm, notification, image-reflector, image-automation) that clone the repo, render Kustomize/Helm, apply to the cluster, correct drift and, optionally, write back to git to bump the tag of a new image. The unit of work is the Kustomization or the HelmRelease, with interval (how often it reconciles), prune (deletes what is no longer in git), dependsOn (orders the start-up) and health checks. The interval sets the MTTR of drift: with interval=1m, a manual edit to the cluster is reverted in ≤ 1 min on average. Secrets cannot sit in the clear in git, a chicken-and-egg problem, and it is solved with SOPS/age, sealed-secrets or External Secrets + Vault. Rollback is a git revert. GitOps is not free: the learning curve is real, and debugging why a Kustomization is not reconciling is a new skill.

The analogy: the master plan and the foreman who does not negotiate

Picture a large building site with a signed and filed master plan, and a foreman who has a single order: the site must be identical to the plan, at all times. The foreman does not improvise. Every so often he walks the site with the plan in hand and compares: if a wall is where the plan says, he leaves it; if someone moved a partition overnight without updating the plan, he puts it back; if the plan says there is a column and it does not exist, he builds it; if a column exists but no longer appears on the plan, he knocks it down. The plan is the single source of truth: to change the site you do not touch the site, you change the plan, and the foreman takes care of the rest on his next round.

That is exactly the mechanics of GitOps. Git is the master plan: declarative (it describes the end state, not the steps), versioned (every change is signed into the history, with author and reason), immutable (a commit is not rewritten). Flux is the foreman: every interval it walks the cluster, compares against git and converges. And here is the lesson that separates GitOps from “keeping the YAML in a repo”: the foreman undoes manual changes. If an operator comes in with kubectl edit and raises the inference engine replicas from 2 to 4 at three in the morning to put out a fire, on the next round Flux takes it back to 2, because the plan says 2. This enrages anyone coming from the imperative world, and it is precisely the point: if you want 4 replicas permanently, you update the plan. The cluster stops being a system with its own opaque memory and becomes a reproducible projection of git. Delete the entire cluster, point a fresh Flux at the same repo, and the site is rebuilt identically.

The analogy also marks the boundary: the plan describes the building, not who holds the keys to the materials store. Secrets (passwords, tokens, keys) cannot go into the public plan. That is the chicken-and-egg problem we cover further down and that the hardening piece develops in full.

The problem: an assistant is six services, not one

Before Flux, it is worth fixing the size of the problem. A minimally serious sovereign LLM assistant, deployed on the generic reference cluster, has this service topology:

Service	Function	Typical namespace	State
Inference engine (vLLM)	Serves the tokens of the general LLM and the code one	`llm-serving`	Stateless (model from object store)
Embeddings + reranker (TEI)	Vectorises queries and reorders candidates for RAG	`llm-serving`	Stateless
Vector store (Qdrant)	Stores and searches document embeddings	`data`	Stateful (PVC)
L7 gateway (Envoy AI Gateway / LiteLLM)	Per-model routing, rate limiting, auth	`gateway`	Semi-stateless
Chat front end	Assistant UI	`apps`	Stateless
Observability (Langfuse, OTel Collector, dashboards)	LLM-aware traces, metrics and logs	`observability`	Stateful (Langfuse Postgres)

Six services, four or five namespaces, two components with persistent state, and a real dependency graph: the front end calls the gateway, the gateway calls the engine and the embeddings service, RAG depends on the vector store being full, and everything emits traces to observability. Deploying this by hand means remembering the order, the values of each helm install, the ConfigMaps, the Secrets, the nodeSelectors that pin the GPU pods to the right nodes. Do it twice (a staging environment and a production one) and the two diverge within days. That is exactly what GitOps eliminates.

The six Flux controllers

Flux is not a monolithic binary but a set of cooperating controllers, the GitOps Toolkit. A default installation brings four; the two image automation ones are added with --components-extra (Flux installation):

source-controller: clones and keeps the sources up to date, whether git repos (GitRepository), Helm repos (HelmRepository), buckets or OCI artifacts. It is the one “holding the plan”: it exposes the repo contents as an internal artifact that the others consume.
kustomize-controller: takes a source artifact, renders Kustomize (bases + overlays) and applies the result to the cluster. It is responsible for prune, dependsOn and the health checks of the Kustomizations.
helm-controller: reconciles HelmRelease objects, installing and updating Helm charts declaratively, without anybody running helm from a terminal (helm-controller).
notification-controller: the bridge to the outside world in both directions. It receives webhooks (to reconcile instantly on every push, instead of waiting for the interval) and emits events and alerts to Slack, an internal chat or an incident system.
image-reflector-controller: scans container registries and stores the tags it finds in an internal database. It is the “eyes” of image automation.
image-automation-controller: uses what the eyes see to write back to git, committing the new image tag into the manifests when a version appears that satisfies the policy.

A cluster without image automation does not need the last two. But for an inference stack where the model server is updated fairly often, they are what automates version promotion without touching anything by hand.

The diagram has three paths. The grey one is the main flow: git → source-controller → kustomize/helm-controller → cluster. The dashed red one is the reconciliation loop that runs every interval: it compares actual state against desired, fixes drift and prunes what is left over. The purple one is image automation: the reflector sees a new tag in the registry, the policy decides whether it qualifies, and the automation-controller writes back to git. That last path is the one that closes the circle and turns git into a system that updates itself.

Repo structure: clusters, infrastructure, apps

The most widespread convention separates three levels of responsibility. It is not the only one, but it ages well:

gitops-repo/
├── clusters/
│ └── prod/
│ ├── infrastructure.yaml # Kustomization → ./infrastructure
│ └── apps.yaml # Kustomization → ./apps (dependsOn infrastructure)
├── infrastructure/
│ ├── controllers/ # ingress, cert-manager, GPU operator, KEDA...
│ └── configs/ # ClusterIssuer, RuntimeClass, StorageClass...
└── apps/
 ├── base/ # manifests common to all environments
 │ ├── llm-engine/ # vLLM HelmRelease
 │ ├── gateway/ # L7 gateway HelmRelease
 │ ├── chat-front/ # Deployment + Service + Ingress
 │ ├── vector-store/ # Qdrant HelmRelease (+ PVC)
 │ └── observability/ # Langfuse + OTel HelmRelease
 ├── staging/ # overlay: low replicas, small model
 └── prod/ # overlay: high replicas, large model, MIG

The key piece is Kustomize’s base / overlay separation. The base/ describes the service once; each overlay (staging/, prod/) applies a patch with the environment’s differences: number of replicas, model size, MIG profile, nodeSelector, --gpu-memory-utilization. That avoids duplicating complete manifests per environment. The clusters/ folder is the entry point Flux reconciles first: it holds the root Kustomizations that point to infrastructure/ and apps/, with a dependsOn that guarantees the infrastructure (CRDs, operators, storage classes) is ready before the applications.

How many manifests: the calculation that justifies overlays

This is where the numbers make the argument. Without overlays, each service needs a complete set of manifests per environment. With $N$ services and $M$ environments, the cost in files to maintain is:

$$ \text{files}_{\text{naïve}} = N \times M \times k $$

where $k$ is the average number of manifests per service (Deployment/HelmRelease + Service + ConfigMap + PVC + Ingress ≈ 5). For our assistant, $N=6$ services and $M=3$ environments (dev, staging, prod), with $k=5$:

$$ 6 \times 3 \times 5 = 90 \text{ complete files, each maintained separately.} $$

With the base/overlay structure, the base is written once and each overlay contains only the patch with the differences (typically 1 short file per service per environment):

$$ \text{files}_{\text{overlay}} = \underbrace{N \times k}_{\text{base}} + \underbrace{N \times M}_{\text{patches}} = 6 \times 5 + 6 \times 3 = 30 + 18 = 48 $$

It is not just that there are fewer files (48 against 90): it is that the bulk of a change happens in a single place. Changing the engine’s memory limit for every environment is one edit in base/, not three synchronised edits. The saving factor grows with $M$: for 5 environments, the naïve version is 150 files and the overlay one is 60. Duplication is the enemy of auditability, and overlays attack it at the root.

The reconciliation loop: desired vs actual

The heart of Flux is a control loop that never ends: every interval it reads the desired state (git), reads the actual state (cluster), computes the difference and applies it. It is the same principle as a thermostat: it reads the target temperature (git), reads the actual temperature (cluster) and switches the boiler on or off until they match. It does not “deploy once”; it converges forever.

A Kustomization manifest for the applications, with the pieces that matter (Kustomization reference):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: apps
 namespace: flux-system
spec:
 interval: 1m # reconciles every minute: sets the drift MTTR
 path: ./apps/prod  # production overlay
 prune: true # deletes from the cluster whatever is removed from git
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 dependsOn:
 - name: infrastructure  # does not apply apps until infra is Ready
 wait: true # waits for the resources to be healthy
 timeout: 5m  # fails the reconciliation if it does not converge in 5 min
 healthChecks:
 - apiVersion: apps/v1
 kind: Deployment
 name: gateway
 namespace: gateway

Four fields do the heavy lifting:

interval defines how often the foreman’s round runs. It is the direct determinant of the drift MTTR (next section).
prune: true switches on garbage collection: objects that were applied before but are no longer in the current git revision are deleted from the cluster automatically. Without prune, a service withdrawn from the repo keeps running, and the plan and the site diverge silently.
dependsOn orders the graph: Flux does not apply the manifests of a Kustomization until every referenced one is in state Ready: True. It is the mechanism that guarantees data → gateway → front end.
wait + healthChecks: with wait: true, Flux monitors every applied resource and waits for them to be ready before marking the reconciliation as successful; healthChecks lets you tune exactly which resources to watch. That is what makes a dependsOn mean something: the dependency is not satisfied until it is healthy, not merely applied.

MTTR and drift detection as a function of the interval

The interval is the only parameter the operator chooses to govern the speed of the loop, and it translates directly into operational metrics. If a deviation (drift) happens at a random instant within the reconciliation period $T$, the waiting time until Flux detects it is uniformly distributed over $[0, T]$, with mean:

$$ \mathbb{E}[t_{\text{detection}}] = \frac{T}{2} $$

To that you add the correction time $t_c$ (render, diff, apply), normally a few seconds. The drift MTTR comes out as:

$$ \text{MTTR}_{\text{drift}} = \frac{T}{2} + t_c $$

With interval=1m and $t_c \approx 10\text{s}$, drift is corrected in $\frac{60}{2} + 10 = 40$ s on average, with a worst case of $60 + 10 = 70$ s. With interval=10m, the mean rises to $5\text{min}\;10\text{s}$ and the worst case to more than 10 min: a window in which a manual change stays active. The temptation is to set interval=10s and forget about it, but there is a cost: every reconciliation consumes CPU and, above all, makes requests to the cluster API and to the registry. With dozens of Kustomizations reconciling every 10 s, the kube-apiserver and the image-reflector start to feel the pressure. The practical rule: a short interval (1m) for critical applications whose drift hurts, a long interval (10–30m) for stable infrastructure that almost never changes, and notification-controller webhooks for instant reconciliation on every push, so that the interval governs only drift and not deployment latency.

Helm as code: the inference engine HelmRelease

For the engine, the gateway and the front end, the usual approach is to package them as Helm charts and declare them with HelmRelease. The helm-controller reconciles them without anyone ever running helm:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: llm-engine
 namespace: llm-serving
spec:
 interval: 10m
 chart:
 spec:
 chart: vllm
 sourceRef:
 kind: HelmRepository
 name: vllm-charts
 values:
 image:
 tag: "0.8.4" # managed by image automation (see below)
 replicaCount: 2
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 resources:
 limits:
 nvidia.com/gpu: 1
 extraArgs:
 - "--gpu-memory-utilization=0.90"
 - "--tensor-parallel-size=1"

Everything that in the imperative world would be a vllm serve flag, --gpu-memory-utilization or --tensor-parallel-size, now lives as versioned data. Changing the fraction of VRAM the engine reserves is a commit, reviewable in a PR, with history. The relationship with sharing a GPU is direct: the MIG profile and the --gpu-memory-utilization are deployment parameters, and here they are code.

Image automation: bumping the tag without touching anything by hand

The inference engine is updated fairly often (vLLM patches, security fixes). Without automation, every new version requires somebody to edit the tag in the HelmRelease. Flux image automation does it on its own, with three objects (automate image updates):

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: vllm
 namespace: flux-system
spec:
 image: registry.example.local/inference/vllm
 interval: 5m  # scans the registry tags every 5 min
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
 name: vllm
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: vllm
 policy:
 semver:
 range: ">=0.8.0 <0.9.0" # only patches and minors within 0.8.x–0.8.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
 name: vllm
 namespace: flux-system
spec:
 interval: 5m
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 git:
 commit:
 author:
 name: fluxbot
 email: fluxbot@example.local
 messageTemplate: "auto: bump vllm to {{ .NewTag }}"

The division of roles: ImageRepository scans all the tags of the image repository and stores them in the reflector’s internal database. ImagePolicy reads those tags and picks the “latest” according to the policy; the policy field is mandatory and defines how the selection is made (ImagePolicy). ImageUpdateAutomation takes the chosen tag, edits the manifest in the repo (wherever there is a # {"$imagepolicy": "flux-system:vllm"} marker) and commits (ImageUpdateAutomation).

The choice of policy matters:

semver: interprets tags as semantic versions and picks the highest one that satisfies the range (range: ">=0.8.0 <0.9.0"). By default it excludes prereleases (0.8.0-rc.1 does not qualify) unless they are asked for explicitly. It is the right option for production: you stay inside a tested version range and do not jump to a new major by accident.
regex / alphabetical / numerical: for tag schemes that are not pure semver, for example main-<sha>-<timestamp> from an internal pipeline. More flexible, but it forces you to trust that tag order reflects version order, which is fragile.

A safety guideline: instead of having ImageUpdateAutomation push directly to main, configure it to write to a branch and open a PR. That way the version bump goes through human review or a validation pipeline before reaching the cluster. Automation proposes; the human (or an automatic gate) disposes. And for production, the version bump is not the deployment: it is the trigger for the progressive rollout described in canary, blue-green and shadow.

Secrets in GitOps: the chicken-and-egg problem

GitOps demands that all the desired state is in git. But secrets, the Langfuse Postgres password, the registry token, the gateway API keys, cannot go into a repo in the clear, not even a private one: git history is immutable, and a secret committed once stays there forever. This is the fundamental tension: GitOps wants everything in git; security forbids secrets in git. There are three families of solution, all with the same underlying idea, that only the encrypted secret goes into git and the cluster holds the key to decrypt it:

SOPS + age/KMS: you encrypt the sensitive values of a manifest with SOPS (it leaves the keys readable and encrypts only the values). The encrypted YAML goes to git; the kustomize-controller carries an age key or reaches a KMS to decrypt it at apply time. Simple, with no extra components in the cluster.
sealed-secrets: a controller in the cluster holds a private key. You encrypt the secret against its public key (with kubeseal), the encrypted SealedSecret goes to git, and the controller decrypts it into a normal Secret inside the cluster. The private key never leaves the cluster.
External Secrets + Vault: not even the encrypted secret goes into git, only a reference (ExternalSecret) saying “the value of this key is in Vault, at this path”. The External Secrets operator resolves it at apply time. It is the cleanest pattern for many secrets and frequent rotation, at the cost of operating a Vault.

The choice depends on the volume of secrets and on whether you already have a central manager. For a small stack, SOPS+age is enough and has no dependencies. The sibling hardening piece goes into the detail of each option, key rotation and the threat model.

Promotion and rollback: git revert is the panic button

The most elegant consequence of having the cluster as a projection of git is that rollback is a git revert. If a deployment breaks production, the new engine gives worse latency or the gateway starts returning 5xx, there is no need to remember which version was there before or to rebuild the state by hand. You revert the commit that introduced the change, and on the next reconciliation (≤ interval, or instantly with a webhook) Flux returns the cluster to the previous state. The git history is, literally, the deployment history: every commit is a restore point with author, date and reason.

This fits with the promotion strategies of the canary, blue-green and shadow post: Flux manages what is declared, and the progressive rollout tool manages how traffic moves between the old version and the new one. The git revert is the last-resort rollback (it returns everything to a known state); the canary is the mechanism that avoids needing it. For services where the version change is delicate, the right approach is for image automation to do the bump on a branch, for the canary to validate the regression gates, and only then to promote the commit to main.

The seams: GitOps is not free

It would be dishonest to sell GitOps as magic. It has real costs that any team pays:

Learning curve. The declarative model is a change of mindset. The operator who has spent years fixing incidents with kubectl edit has to unlearn the reflex: now the cluster reverts their changes and that, at first, feels like Flux “fighting them”. Understanding that the change is made in git, not in the cluster, takes weeks of discomfort.

Debugging the reconciler. When a Kustomization does not apply, the error is not in the pod; it is in the reconciliation chain. Did the source-controller clone the right revision? Did the kustomize-controller render the overlay properly? Did a dependsOn health check fail? Did the timeout fire before the large model finished loading? Diagnosing this needs flux get, flux logs, reading the status.conditions of the Flux objects and understanding which link got stuck. It is a new skill, different from debugging plain Kubernetes.

Timeouts and slow loads. The timeout of a Kustomization with wait: true has to be longer than the start-up time of the slowest service. An inference engine that takes 4 minutes to load a large model from the object store will fail a Kustomization with timeout: 2m, even though everything is fine. Calibrating timeouts per service is fine-grained work.

The drift you did want. Sometimes the operator needs an urgent temporary change and Flux reverts it. The correct answer, suspending reconciliation of that Kustomization with flux suspend, making the change, then reflecting it in git and resuming, is discipline that has to be built. Without that discipline, people end up fighting the foreman at 3 in the morning.

None of these costs cancels the benefit. But a team adopting GitOps expecting “everything to be easier from day one” gets frustrated. It is easier from month two onwards, once auditability, reproducibility and trivial rollback have paid off the curve.

Applied to the generic 4×H100 cluster

On the reference cluster (4×H100 SXM 80 GB per GPU node, NVLink, plus CPU and control nodes), the assistant stack is managed entirely by GitOps. Hardware decisions become data in the overlays.

nodeSelectors and MIG profiles as code. The prod overlay pins each service to the right node and declares the GPU profile. The general engine uses a whole GPU; embeddings and the small LLM fit in MIG slices, exactly the split from the sharing a GPU post, but now versioned:

# apps/prod/llm-engine-patch.yaml
spec:
 values:
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 extraArgs:
 - "--gpu-memory-utilization=0.92"
 - "--tensor-parallel-size=1"
---
# apps/prod/embeddings-patch.yaml — on a MIG slice
spec:
 values:
 nodeSelector:
 nvidia.com/mig.config: "3g.40gb"
 resources:
 limits:
 nvidia.com/mig-3g.40gb: 1

The MIG profile (3g.40gb), the VRAM fraction (0.92), the tensor-parallel setting: all of it is text in a PR. Changing the GPU split between services is a reviewable commit, not a hand-run nvidia-smi mig session that nobody records.

Start-up order with dependsOn. The assistant’s dependency graph is encoded with chained dependsOn. The right order is data → embeddings/engine → gateway → front end:

# apps/prod/gateway.yaml
spec:
 dependsOn:
 - name: llm-engine  # the gateway is useless without the engine
 - name: embeddings
 # ...
---
# apps/prod/chat-front.yaml
spec:
 dependsOn:
 - name: gateway  # the front end is useless without the gateway

With wait: true on each Kustomization, Flux does not mark llm-engine as Ready until the engine pod passes its health check, which in the case of vLLM means model loaded and endpoint responding, not just pod started. Only then does it begin applying the gateway. This avoids the classic cascade of failures of manual deployment: bring up the front end first, see a 502 because the gateway is not there, bring up the gateway, see a 503 because the engine is still loading the model. With dependsOn + wait, the order is guaranteed by the reconciler, not by the operator’s memory.

The result: deleting the assistant’s entire namespace and letting Flux rebuild it from git produces exactly the same stack, in the same order, with the same GPU configuration. The 4×H100 cluster stops having a configuration known only to whoever built it, and becomes a reproducible projection of a repo that anyone can audit.

Conclusion

GitOps is not a tool, it is an inversion of the direction of control: instead of pushing changes to the cluster, you declare the state in git and let an agent pull the cluster towards it. For an LLM assistant (six services, several namespaces, start-up dependencies, delicate GPU configuration) that inversion turns a fragile and opaque system into a reproducible and auditable one. Flux is the foreman who keeps the site identical to the plan and undoes any change that has not gone through the plan first. The price is a real learning curve and a new debugging skill. The prize is that the cluster stops keeping secrets about itself: everything it is, is written down, signed and reproducible in git.

References

OpenGitOps — GitOps principles — opengitops.dev
Flux installation — fluxcd.io/flux/installation
Flux Kustomization — fluxcd.io/flux/components/kustomize/kustomizations
helm-controller — github.com/fluxcd/helm-controller
Flux Image Policies — fluxcd.io/flux/components/image/imagepolicies
Flux Image Update Automations — fluxcd.io/flux/components/image/imageupdateautomations
Automate image updates to Git — fluxcd.io/flux/guides/image-update
SOPS — github.com/getsops/sops

Helm on lo0 — Tech Blog

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

TL;DR

Quality ≠ performance: the independent Pareto axis

Reference benchmarks: what each one measures

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

HELM (Stanford CRFM)

LiveBench

lighteval (Hugging Face)

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

LMArena / Chatbot Arena

Decision table: which tool for which objective

The data contamination trap

What it is

Scale of the problem (2024–2025 data)

Detection mechanisms

Why static leaderboards saturate

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

Few-shot vs zero-shot: the numerical difference

Prompt sensitivity

Why two evals give different results

The confidence interval formula for comparing two models

The quality axis in the scorecard

See also

Sources

GitOps for the inference stack with Flux: operating the assistant as code

TL;DR

The analogy: the master plan and the foreman who does not negotiate

The problem: an assistant is six services, not one

The six Flux controllers

Repo structure: clusters, infrastructure, apps

How many manifests: the calculation that justifies overlays

The reconciliation loop: desired vs actual

MTTR and drift detection as a function of the interval

Helm as code: the inference engine HelmRelease

Image automation: bumping the tag without touching anything by hand

Secrets in GitOps: the chicken-and-egg problem

Promotion and rollback: git revert is the panic button

The seams: GitOps is not free

Applied to the generic 4×H100 cluster

Conclusion

See also

References