Llm-as-Judge on lo0 — Tech Blog

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

Thu, 04 Jun 2026 09:00:00 +0200

TL;DR

A RAG pipeline fails in ways that user satisfaction cannot distinguish: the LLM can hallucinate even with good chunks, or retrieval can ignore key documents even though the LLM synthesises well what it receives. RAGAS breaks evaluation down into four orthogonal metrics, faithfulness, answer relevance, context precision and context recall, each pointing at a different sub-component. The golden dataset is the reference calibrator; without it the metrics have no anchor. The complete stack runs 100 % on-premise with vLLM as the judge and Langfuse for traceability.

The master analogy: the quality inspector at a furniture factory

Imagine you manufacture chairs. You could ask customers “is it comfortable?” and leave it there. But that question does not tell you what to fix when the answer is “no”. The quality inspector does not ask that: they measure the board with a Shore hardness test, check that each leg is exactly 45 cm, verify that the assembly manual includes the twelve screws from the BOM, and detect whether a low-density board slipped past the incoming filter.

RAGAS is that inspector applied to RAG:

Faithfulness → does the board have the specified hardness? The LLM can only use the material (chunks) that retrieval hands it.
Context Precision → is the leg exactly the right length? Of the K retrieved chunks, how many are genuinely useful and how many are filler that confuses the assembler?
Context Recall → does the manual include all the screws? Of all the facts the correct answer should contain, how many appear in the retrieved chunks?
Noise Sensitivity → if the operator uses a medium-low density board, does it show in the final product? If you introduce irrelevant chunks, does the LLM start hallucinating?

Without measuring each dimension separately, the diagnosis is opaque: “the RAG is not working well” does not tell you whether to repair the embedder, the reranker, the prompt or the corpus.

The problem of evaluating RAG

Classification has one uncomfortable virtue: if you predict 87 out of 100 labels correctly, accuracy = 0.87. There is no ambiguity. RAG does not enjoy that luxury.

A RAG system can fail in at least three independent ways:

Retrieval correct, LLM hallucinates: the chunks contain the correct answer, but the LLM generates claims that are not in those chunks. Faithfulness low; context recall high.
LLM correct, retrieval fails: retrieval returns irrelevant chunks (low context precision) or incomplete ones (low context recall). If the LLM has enough parametric knowledge, it may look as though it is answering well, but in reality it is ignoring the context, which is a time bomb once the parametric knowledge goes stale.
Retrieval and LLM both correct, answer does not answer the question: the answer is faithful to the context and the chunks are relevant, but the question was a different one. Answer relevance low.

Each failure mode requires a different metric and a different corrective action. Using a single metric (BLEU, ROUGE, user satisfaction) mixes the signals and makes it impossible to prioritise improvement work.

The four RAGAS metrics

1. Faithfulness — fidelity to the context

Question: how many claims in the generated answer are supported by the retrieved chunks?

Calculation:

$$\text{Faithfulness} = \frac{|\text{claims supported by the context}|}{|\text{total claims in the answer}|}$$

The process uses an LLM-as-judge (see https://blog.lo0.es/en/posts/llm-as-judge-exam-marker-not-oracle/): first the atomic claims are extracted from the answer (“the model was released in 2023”, “it supports contexts of 128k tokens”, and so on), then the judge classifies each claim as supported or not supported by the chunks.

Example: The generated answer has 5 claims. The judge determines that 4 are in the chunks and 1 is an unsupported extrapolation.

$$\text{Faithfulness} = \frac{4}{5} = 0{.}80$$

Warning sign: faithfulness < 0.85 indicates that the LLM is generating content that goes beyond the context, which is to say it is hallucinating with superficial backing.

2. Answer Relevance — relevance of the answer

Question: does the answer actually answer the question as posed?

Intuition: An answer that answers the question well “implies” that question. If you generate N hypothetical questions from the answer and measure their semantic similarity to the original question, you get a relevance signal.

Calculation:

$$\text{AnswerRelevance} = \frac{1}{N} \sum_{i=1}^{N} \cos(\vec{q}_{\text{original}}, \vec{q}_{i}^{\text{generated}})$$

where $\vec{q}$ are question embeddings.

Example: For the question “Which Python versions does FastAPI support?” and an answer about web frameworks in general, the hypothetical questions generated will be along the lines of “what are the best web frameworks?”, with low cosine against the original question → low answer relevance.

3. Context Precision — retrieval precision

Question: of the K retrieved chunks, what proportion are genuinely relevant?

Calculation (weighted version):

$$\text{ContextPrecision@K} = \frac{\sum_{k=1}^{K} \text{Precision@}k \cdot \mathbb{1}[\text{chunk}_k \text{ is relevant}]}{|\text{relevant chunks in top-K}|}$$

The most direct form: the LLM judge classifies each chunk as relevant or not for answering the question. Precision is the relevant fraction.

Example: 5 chunks are retrieved. The judge considers 3 relevant and 2 noise.

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

Warning sign: precision < 0.6 indicates that retrieval is contaminating the context with information that can contradict or dilute the correct answer.

4. Context Recall — retrieval recall

Question: of all the facts needed to build the correct answer (ground truth), what proportion are covered by the retrieved chunks?

Calculation:

$$\text{ContextRecall} = \frac{|\text{ground-truth claims attributable to some chunk}|}{|\text{total claims in ground truth}|}$$

This metric requires ground truth, which is to say you need the golden dataset.

Example: The ground truth has 6 claims. The retrieved chunks cover 5 of them; the sixth is in a document that retrieval did not find.

$$\text{ContextRecall} = \frac{5}{6} \approx 0{.}83$$

5. Noise Sensitivity — sensitivity to noise

Question: if you introduce irrelevant chunks into the context, how far does faithfulness fall?

It is computed as the difference in faithfulness between the clean condition and the condition with controlled noise:

$$\Delta\text{Faithfulness} = \text{Faithfulness}_{\text{clean}} - \text{Faithfulness}_{\text{noisy}}$$

A $\Delta > 0{.}15$ indicates that the prompt or the LLM are fragile against contaminated contexts, a problem that low context precision turns into a routine occurrence.

A complete worked example

Scenario

Question: “What latency guarantees does Redis offer for in-memory read operations with a 10 GB dataset?”

Answer generated by the RAG:

“Redis guarantees sub-millisecond latencies for GET operations on datasets that fit in RAM. With a 10 GB dataset and clients on the same local network, P99 is typically below 1 ms. Redis also supports horizontal clustering to scale beyond the RAM of a single node. In addition, Redis has included native JSON support since version 7.2.”

Ground truth (reference answer):

“Redis operates entirely in memory, which guarantees sub-millisecond latencies for GET. On local networks with 10 GB datasets in RAM, P99 sits below 1 ms. Clustering allows scaling beyond the RAM of a single node.”

Retrieved chunks (5 chunks, summarised fragments):

#	Summarised content	Relevant
C1	“Redis operates in memory; GET has latencies < 1 ms on LAN”	Yes
C2	“Redis Cluster allows sharding to scale total RAM”	Yes
C3	“Redis Sentinel manages high availability through automatic failover”	No
C4	“Redis benchmarks: P50 = 0.3 ms, P99 = 0.9 ms on a 10 GB dataset”	Yes
C5	“Redis Stack adds modules: RedisJSON, RediSearch, RedisTimeSeries”	No

Step-by-step calculation

Faithfulness:

Claims in the generated answer:

“Redis guarantees sub-millisecond latencies for GET on in-RAM datasets” → supported by C1, C4
“With 10 GB on LAN, P99 < 1 ms” → supported by C4
“Redis supports horizontal clustering to scale RAM” → supported by C2
“Redis has included native JSON support since version 7.2” → NOT supported by any chunk (C5 mentions RedisJSON as a Redis Stack module, not as native to Redis core)

$$\text{Faithfulness} = \frac{3}{4} = 0{.}75$$

Claim 4 is an extrapolation that mixes information from C5 imprecisely, a partial hallucination.

Context Precision:

Relevant chunks: C1, C2, C4 (3 out of 5).

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

C3 and C5 are noise. C5 in particular contributed to the partial hallucination about JSON.

Context Recall:

Ground-truth claims:

“Redis operates in memory, GET < 1 ms” → attributable to C1 ✓
“P99 < 1 ms on LAN with 10 GB” → attributable to C4 ✓
“Clustering scales beyond the RAM of a single node” → attributable to C2 ✓

$$\text{ContextRecall} = \frac{3}{3} = 1{.}00$$

Retrieval found every chunk needed for the ground truth. The problem is not recall but precision (C3 and C5 contaminated the context).

Answer Relevance:

The judge generates 3 hypothetical questions from the answer:

“What latencies does Redis offer for in-memory reads?” — cos = 0.91
“How does Redis scale horizontally?” — cos = 0.74
“Which JSON modules does Redis include?” — cos = 0.52 (a drift caused by the hallucination)

$$\text{AnswerRelevance} = \frac{0{.}91 + 0{.}74 + 0{.}52}{3} = 0{.}72$$

The drift towards JSON reduced relevance. A tighter answer would have scored around 0.90.

Summary of the example

Metric	Value	Diagnosis
Faithfulness	0.75	LLM extrapolated beyond the context
Context Precision	0.60	Retrieval returned 2 irrelevant chunks
Context Recall	1.00	Retrieval captured everything needed
Answer Relevance	0.72	Answer drifts off topic

Main corrective action: improve the reranker so that C3 and C5 are filtered before they reach the LLM. The faithfulness and relevance problem is a direct consequence of the low precision, not of the LLM itself.

Building the golden dataset

What it is and why it matters

The golden dataset is a set of (question, relevant chunks, correct answer) tuples that acts as a reference calibrator. Without it, context recall cannot be computed (there is no ground truth) and the other metrics lack any interpretive anchor: is 0.75 faithfulness good or bad for this corpus and this domain?

A well-built golden dataset lets you:

Compare pipeline versions (embedder v1 vs v2, chunk size 512 vs 1024)
Detect regressions in CI before deploying
Stratify the analysis by question type

LLM-assisted construction pipeline

Purely manual construction is expensive. The standard pattern in 2026 is LLM assistance with human review of a sample:

Step 1 — Seed chunk selection. From the full corpus, select representative chunks through stratified sampling (by section, date, document type). For a technical corpus of 10,000 chunks, 500-1,000 seeds is a reasonable starting point.

Step 2 — Question generation. A powerful LLM (Llama-3.1-70B or similar) generates 2-3 questions per seed chunk using a prompt along these lines:

Given the following documentation fragment, generate specific questions
that can only be answered correctly using THIS fragment and not
general knowledge. The questions should be the ones an engineer looking
for operational information would ask.
Fragment: {chunk}

Step 3 — Reference answer generation. The same LLM, with access to the seed chunk (and to adjacent chunks if the question requires them), generates the reference answer.

Step 4 — Human review of a sample. Manually review 10-20 % of the generated dataset. The most common rejection criteria: trivial questions that any LLM answers without the corpus, answers the LLM padded with parametric knowledge instead of the chunks, and badly formulated or ambiguous questions.

Minimum size

Use case	Minimum pairs	Notes
Prototype / initial validation	50-100	Enough to spot coarse problems
Technical corpus in production	200-500	Allows basic stratification
Robust production with full stratification	500-1,000+	Needed to detect subtle regressions

Stratifying the dataset

A flat golden dataset measures the average but hides the extreme cases. The minimum recommended stratification includes three question types:

Easy (single-hop): A single chunk contains all the information needed. The baseline that any decent RAG must beat.
Hard (multi-hop): The correct answer requires combining information from 2-4 different chunks. This is where the limits of the reranker and the synthesis prompt show up.
Adversarial: The question has a false premise, or the corpus does not contain the answer. A correct RAG must answer “I do not have enough information”, whereas a fragile RAG hallucinates confidently. This question type directly measures the risk of high-impact hallucination.

The Goodhart trap

“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart

If you optimise the embedder or the reranker using the golden dataset as a loss function, the dataset is corrupted as a metric: the system learns to perform well on those specific questions without improving on the general domain.

The solution is the same as in supervised ML: separate the dev set (for optimisation and iteration) from the test set (for final evaluation, frozen and audited). The test set must never be used to make design decisions; only to report the state of the system at release time.

Correlation with real satisfaction

Field studies published by the Databricks teams (2024) and the RAGAS adoption analyses (2025) point to interpretable operational thresholds:

Metric range	Observable symptom	Corrective action
Faithfulness < 0.75	Users frequently report “made-up answers”	Review the LLM prompt; increase citation instructions; lower the temperature
Faithfulness 0.75-0.85	Occasional hallucinations on peripheral topics	Improve context precision to eliminate contaminating chunks
Faithfulness ≥ 0.85	Correlates with positive NPS in field studies	Hold steady; monitor for drift
Context Precision < 0.60	LLM includes contradictory information; inconsistent answers	Tune the reranker; reduce K; review similarity thresholds
Context Recall < 0.70	Multi-hop questions fail; key information missing	Review the chunking strategy; add larger chunks; enrich metadata
Answer Relevance < 0.70	Answers that are “correct but do not answer”	Review the synthesis prompt; add an explicit instruction to stick to the question

Low context precision is especially pernicious: irrelevant chunks are not neutral. They raise the probability that the LLM uses incorrect information as though it were relevant, degrading faithfulness in a chain reaction. It is the transmission belt by which a retrieval problem turns into an LLM problem.

Diagram: the continuous evaluation loop

The continuous evaluation loop: corpus → retrieval → LLM → RAGAS judge → metrics → alert → corrective action → corpus.

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

The ragas library supports asynchronous evaluation and multiple LLM backends. Integrating vLLM as the judge removes any need to send data to external APIs, which is critical in environments with sensitive data.

from ragas import evaluate
from ragas.metrics import (
 faithfulness,
 answer_relevancy,
 context_precision,
 context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Judge LLM pointing at on-premise vLLM
judge_llm = ChatOpenAI(
 model="meta-llama/Llama-3.1-70B-Instruct",
 base_url="http://vllm-service:8000/v1",
 api_key="sk-local", # vLLM ignores the value but requires the field
)

embeddings = OpenAIEmbeddings(
 model="BAAI/bge-m3",
 base_url="http://embedding-service:8001/v1",
 api_key="sk-local",
)

result = evaluate(
 dataset=golden_dataset, # HuggingFace Dataset with standard columns
 metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
 llm=judge_llm,
 embeddings=embeddings,
)

The dataset RAGAS expects has four columns: question, answer, contexts (a list of strings), ground_truth.

Langfuse for eval traceability

Each RAGAS evaluation is recorded in Langfuse as a dataset experiment, linking the scores to the production spans (see https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/). This makes it possible to correlate a drop in faithfulness with the specific request that caused it. Without that link, the metrics are numbers with no actionable context.

from langfuse import Langfuse

lf = Langfuse()

# Create or retrieve the dataset in Langfuse
dataset = lf.get_or_create_dataset("rag-golden-v3")

# Record the experiment scores
for idx, row in result.to_pandas().iterrows():
 lf.score(
 name="ragas-faithfulness",
 value=row["faithfulness"],
 trace_id=row["trace_id"], # linked to the production span
 )

Prometheus + Grafana for operational alerts

RAGAS metrics are exposed as Prometheus gauges. A Grafana dashboard with thresholds configures alerts when faithfulness falls sustainedly below 0.80:

# Prometheus alert rule
- alert: RAGFaithfulnessLow
 expr: avg_over_time(rag_faithfulness_score[30m]) < 0.80
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "RAG faithfulness below threshold ({{ $value | humanize }})"
 description: "Review context precision and the reranker. Possible corpus drift."

Running RAGAS against on-premise vLLM — practical considerations

Judge size: Llama-3.1-70B as the judge produces results comparable to GPT-4 on faithfulness and context evaluation, according to the RAGAS 0.2 benchmarks (2025). Smaller models (8B-13B) degrade judge quality on multi-hop questions.
Throughput: On on-premise hardware with 4×H100 SXM (320 GB, NVLink), a run of 200 evaluations with Llama-3.1-70B takes roughly 8-12 minutes with batch_size=8 and vLLM in continuous batching mode.
Cost per evaluation: With no external API, the marginal cost is electricity plus GPU amortisation. With 4×H100 at around 3 kW sustained, a run of 200 evaluations costs < 0.10 € in energy at a typical industrial tariff.
Recommended frequency: weekly offline eval over the full golden dataset plus sampled online eval (5-10 % of production requests) with a subset of metrics that do not require ground truth (faithfulness, answer relevance).

What we have not covered

Alternatives to RAGAS: TruLens (evaluation with modular feedbacks), DeepEval (programmatic assertions, pytest integration), ARES (Stanford framework with trained classifiers instead of LLM-as-judge), and the OpenAI evals framework. Each has different trade-offs in judge cost, reliability and ease of integration.
Continuous eval in production: automatically sampling real requests, anonymising them, running a subset of metrics without ground truth and using the result to detect system drift before users report it. It requires a data pipeline separate from the inference pipeline.
Multilingual eval: RAGAS with a judge in Spanish or Catalan over a non-English corpus has documented biases when the judge is a model trained fundamentally in English. The semantic similarity embeddings for answer relevance are especially sensitive to corpus language vs judge language.
A/B testing of RAG configurations: using RAGAS metrics as the success criterion in controlled experiments, chunk size 512 vs 1024, pure BM25 vs hybrid, cross-encoder reranker vs biencoder, with statistical significance computed over the golden dataset.

References

Es Shahul, et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 (2023). https://arxiv.org/abs/2309.15217
RAGAS Documentation v0.2. Metrics Reference. https://docs.ragas.io/en/stable/concepts/metrics/ (accessed June 2026)
Langfuse. Dataset Experiments. https://langfuse.com/docs/datasets/overview (accessed June 2026)
Databricks. LLM Quality Evaluation: From Lab to Production. Databricks Engineering Blog (2024).
Saad-Falcon, J. et al. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476 (2023).
Goodhart, C.A.E. Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics. Reserve Bank of Australia (1975). Modern formulation of the law that bears his name.
vLLM Project. OpenAI-Compatible Server. https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html (accessed June 2026)

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

Mon, 01 Jun 2026 16:30:00 +0200

This post complements those on LLM autoscaling on Kubernetes (the autoscaler coexists with the rollout and must respect it), GPU observability for LLM inference (the metrics that act as gates come from there), Evals for LLMs (the eval that decides whether the new model is ready), LLM-as-judge (the technique that puts the “quality” into the canary gate) and Retrain: closing the loop (the previous step that the new model comes out of).

TL;DR

Promoting a new version of an LLM model to the production cluster without cutting traffic or breaking the SLO demands progressive deployment. The three canonical strategies, blue-green, canary and shadow, answer different questions and have different costs. Blue-green: a complete new pool brought up in parallel, atomic load balancer switch. Instant rollback (point back at the old pool); it demands twice the GPUs during the window. Canary: traffic is shared progressively between the old version and the new one (1 % → 5 % → 25 % → 100 %), measuring regression gates at each step; it consumes incrementally less hardware but exposes real users to the new model from the first percentage point. Shadow / mirror: the old model serves 100 % of the real traffic to the client and, in parallel, a copy of every request goes to the new model without returning its response to the user; it isolates you from quality risk but spends the new model’s GPU on responses nobody consumes, and it does not work well with long SSE streaming. The choice depends on three factors: available GPU budget, service criticality and the availability of a fast automatic eval. The five regression metrics any LLM canary should gate on are: TTFT P95, error rate (HTTP 5xx + premature finish_reason="length"), quality score with LLM-as-judge over a golden set, statistical drift of output embeddings (Wasserstein or KL against the baseline distribution) and cost per request (tokens/s and kW/request). On Kubernetes, Argo Rollouts manages traffic and the AnalysisTemplates as automatic gates; Flagger is the more opinionated alternative. vLLM v1 does not support hot model swap robustly as of May 2026, so the rollout unit is the whole replica (deployment v2 alongside deployment v1). The three specific pitfalls: LB sticky sessions break the statistical comparability of the canary (a client A always lands on the new one, B on the old one, so the populations are not equivalent); semantic eval with LLM-as-judge takes 2–8 seconds per sample and is no use as a real-time gate (it is used in post-analysis or offline before promotion); SSE streaming complicates shadowing because the new model’s response has to be discarded without affecting the old one’s. This post includes a minimal Argo Rollouts manifest applicable to a generic cluster with the NVIDIA GPU Operator.

You are here: DEPLOY (and the transition to RETRAIN)

A new model does not appear in the cluster by magic: it comes from the retrain loop or from a weights provider update. The step between “I have an artifact that passed offline eval” and “it is serving 100 % of the traffic” is exactly this post.

The analogy: opening night of a play

A theatre company is about to premiere a new version of a play that has been running successfully for a year. The company knows several hard facts: the current audience pays for a consistent experience; a bad opening damages the business for months; but premiering nothing leaves the company obsolete against the competition.

The three opening routes the management can choose are the same three of the LLM rollout.

Dress rehearsal behind closed doors (shadow / mirror). The new actors perform the whole play to an empty theatre. There is no audience; nobody buys a ticket. Three full run-throughs are enough to check continuity, timing and the chemistry of the cast. It is expensive because there are salaries and theatre rent, but it does not expose the audience to risk. Useful when the new cast is untested and the director wants to see how it holds up over a complete performance before selling it. In LLM terms: the new model processes every real request in parallel with the old one but its responses are discarded; you spend the new model’s GPU on responses nobody sees.

Cast by performance, alternating (canary). Instead of changing the whole cast at once, Thursday performances belong to the new cast, Friday ones to the old one, Saturday ones half and half. The management reads the visitors’ book comments and the audience numbers performance by performance, deciding after two weeks whether to promote the new cast to permanent or withdraw it. Cheaper than the dress rehearsal because the performances still sell tickets, but it exposes a real audience to risk from the first Thursday. In LLM terms: traffic is shared progressively between the old version and the new one, measuring gates at each step.

Double company with an atomic switch (blue-green). The company hires the new cast, prepares it for a month behind closed doors, and one Saturday announces: “from the next opening night every performance is with the new cast”. If the first performance goes badly, they go back to the old cast in 24 hours, but during that month of preparation double salaries are paid to both companies. In LLM terms: two complete pools of the same size, instant LB switch from one to the other, rollback in seconds if the metrics break.

The analogy also supports the decision: the choice depends on how critical the play is for the business (criticality of the LLM service), how much budget there is to sustain two casts at once (GPU budget), and how much confidence there is in the new cast from the chamber rehearsals (offline eval prior to the canary).

The three strategies in detail

Blue-green

The operator keeps two replica pools identical in size: the blue one (production version v1) and the green one (candidate version v2). When v2 is validated offline (eval passed, smoke tests), the LoadBalancer switch redirects 100 % of the traffic from blue to green in a single step. If the SLO metrics break, the switch goes back in seconds.

Cost: 2× GPUs for the whole window (preparing v2 + the post-switch observation window). For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), preparing the blue-green needs 16 extra GPUs for 1–3 days.

Risk: the switch is atomic. If v2 has a problem that did not show up in offline eval but does show up at scale (for example, edge cases that only appear at 200 RPS), 100 % of users notice it at once. Rollback is instant, but the requests of the first minute after the switch were already affected. So blue-green is preferable when you have high confidence in v2 (a minor change: same architecture, same format, just a new version of the weights) and you prioritise immediate rollback over gradual exposure.

Canary

The operator deploys v2 with a small number of replicas (typically 1) next to the v1 pool. The LoadBalancer shares the traffic out progressively on a schedule: 1 % for 30 minutes → 5 % for 1 hour → 25 % for 2 hours → 50 % for 4 hours → 100 %. Between each step, an analysis gate evaluates regression metrics over the traffic that is already landing on v2. If the gate fails, the rollback withdraws traffic from v2 automatically and leaves v1 serving everything.

Cost: incremental. At the start (1 % of traffic) one v2 replica is enough; at 50 % you need half as many v2 replicas as v1 has in total. Peak extra GPU during the canary: about 30–50 % above baseline.

Risk: real users are seeing v2 from the first 1 %. If v2 produces responses with degraded quality but normal TTFT and error rate, the affected users perceive the degradation without the gate detecting it (unless the gate includes quality drift, which is slow). So canary is preferable when you have medium confidence in v2 (a significant change: different architecture or training) and you accept that a low percentage of users will be guinea pigs.

Shadow / mirror

The LoadBalancer sends 100 % of the real traffic to v1 (which answers the client) and duplicates every request towards v2 (whose response is discarded or stored for offline analysis). The client never sees v2; it is never exposed to the risk.

Cost: 100 % extra compute for v2 with no direct user value for the whole shadow window. For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), a shadow of the same size consumes 16 extra GPUs full time.

Risk: shadow is the safest for the user. But it has two serious limitations: (a) if v2 has a bottleneck that makes the copied request to the shadow take a long time, the shadowing proxy can consume LB connections; it must be out-of-band (asynchronous); (b) long SSE streaming complicates mirroring because you have to keep two parallel streams and discard one while the other flows to the client. Common pattern: shadow only non-streaming requests (short completions, classification), manual offline eval of the streaming requests.

The five regression metrics that act as a gate

Without automatic gates, the “canary” is just a nice name for “manual rollout with a variable percentage”. The gates are the piece that turns the canary into a defensible operation.

Metric 1 — TTFT P95. Comparison of the new model’s P95 against the baseline (v1) P95 in 5-minute windows. Gate: ttft_p95(v2) / ttft_p95(v1) < 1.10. It detects prefill latency regressions (a slower new model) or engine problems (suboptimal config). Source: vllm:time_to_first_token_seconds_bucket (see GPU observability for LLM inference).

Metric 2 — Error rate. Sum of HTTP 5xx + unexpected 4xx + rate of premature finish_reason="length" (responses cut short because the new model does not generate EOS). Gate: error_rate(v2) - error_rate(v1) < 0.01 (1 percentage point). It detects engine crashes, a broken tokenizer, generation problems. Source: vllm:request_success_total{status=...}.

Metric 3 — Quality score (LLM-as-judge). Over a golden set of 200–1,000 representative prompts, v1 and v2 are run offline and a judge model (typically larger: GPT-4 class, Claude, local Llama 405B) scores each pair. Typical gate: mean_score(v2) >= mean_score(v1) - 0.05. This metric is not measured in real time during the canary: judge inference takes 2–8 seconds per sample and does not scale as an inline gate. It is used as an offline pre-promotion gate (before starting the canary) and as a post-mortem over a sample of real traffic captured during the canary. See LLM-as-judge for the mechanics.

Metric 4 — Statistical output drift. For every request that lands on v2 during the canary, embed the response with a lightweight embedding model (e5, BGE) and compare the distribution of v2 embeddings against the distribution of the v1 baseline over the same window. Usable metrics: Wasserstein distance, KL divergence, or more simply, comparing means and variances per dimension. Gate: normalised distance < a calibrated threshold (typically Wasserstein < 0.15). It detects subtle changes in style, length and vocabulary that LLM-as-judge does not capture without going through it too. It is fast: the lightweight embedding takes around 50 ms per response.

Metric 5 — Cost per request. Output tokens per request and kW per request. Gate: cost_per_request(v2) / cost_per_request(v1) < 1.20. It detects new models that generate significantly longer responses or that consume more energy for the same load (quantisation degradation, optimisations failing). Without this gate, an “update” can silently double the bill.

Metric	Type	Measurement latency	Typical gate	Detection
TTFT P95	Quantitative	5 min	`< 110% baseline`	Latency regression
Error rate	Quantitative	1 min	`< 1pp over baseline`	Crashes, generation broken
Quality (LLM-judge)	Offline semantic	hours, over golden set	`> baseline − 0.05`	Functional quality
Statistical drift	Statistical	~5 min	Wasserstein < 0.15	Style, length, vocabulary
Cost per request	Quantitative	5 min	`< 120% baseline`	Economic/energy efficiency

The mechanics on Kubernetes: Argo Rollouts

Argo Rollouts extends the standard Kubernetes Deployment with a new Rollout resource that orchestrates traffic progression and automatic analyses. It integrates with any service mesh (Istio, Linkerd) or ingress controller that supports traffic splitting (NGINX, Traefik, Gateway API).

A minimal example of a 1 → 5 → 25 → 100 % canary with TTFT and error rate gates:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama70b }
spec:
 replicas: 4
 strategy:
 canary:
 canaryService: vllm-llama70b-canary
 stableService: vllm-llama70b-stable
 trafficRouting:
 nginx:
 stableIngress: vllm-llama70b-ingress
 steps:
 - setWeight: 1
 - pause: { duration: 30m }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 5
 - pause: { duration: 1h }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 25
 - pause: { duration: 2h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 50
 - pause: { duration: 4h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 100
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata: { labels: { app: vllm-llama70b } }
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args: [ --model=/models/llama-70b-fp8-v2 ]  # new version
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: ttft-error-gate }
spec:
 metrics:
 - name: ttft-p95-ratio
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 1.10
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 - name: error-rate-diff
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 0.01
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 sum(rate(vllm:request_total{version="v2",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v2"}[5m]))
 -
 sum(rate(vllm:request_total{version="v1",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v1"}[5m]))

If any of the AnalysisTemplates fails, Argo Rollouts backs out automatically: it sets weight=0 on the canary, alerts the operator and keeps v1 serving 100 %. Human operation is reduced to investigating the failure and deciding whether to relaunch or abort.

Flagger offers a more opinionated alternative: weight progression is automatic as a function of metric success instead of a fixed pause; the operator defines a target (maxWeight: 100, stepWeight: 10, metrics: [...]) and Flagger raises or lowers it according to behaviour. Both are mature as of May 2026; the choice is usually dictated by which service mesh is already in the cluster.

The vLLM detail: why there is no “hot swap” of the model

As of May 2026, vLLM v1 does not support hot swapping the model inside the same replica without restarting the engine. The --model argument is evaluated at start-up; changing it requires re-instantiating the LLMEngine, which restarts connections and discards the KV cache. So the rollout unit is the whole replica: you do not do “v1 loads the new model on one of its GPUs” but “a v2 replica is brought up next to a v1 replica and traffic is shared out via the LB”.

TensorRT-LLM with Triton has a similar mechanism: changing the model requires a reload of the Triton backend. SGLang does not support robust hot swap either. The operational consequence: LLM rollout is always going to cost extra GPUs during the window, and the choice between blue-green, canary and shadow is exactly the question of how many extra and for how long.

The three pitfalls specific to LLM rollout

Pitfall 1 — sticky sessions break canary comparability. If the LoadBalancer does session affinity by client IP (common in NGINX and Traefik with loadbalancer.kubernetes.io/session-affinity: ClientIP), a user A always lands on v2 while B always lands on v1. Load distributions, prompt profiles and client behaviour are not random between the two pools, which statistically invalidates any gate comparison. Solution: for a canary, disable session affinity (sessionAffinity: None) or use affinity by random request-id. If the client app requires stickiness for functional reasons (conversational memory persisted in cache), canary is not the right strategy; use blue-green or shadow.

Pitfall 2 — LLM-as-judge is not an inline real-time gate. The temptation to use quality score as a live gate is strong, but the judge’s latency (2–8 s per sample) makes it unfeasible to evaluate more than a 1–2 % sampling of the traffic, and the results arrive minutes late. Operational solutions: (a) offline pre-canary eval over a golden set as a prerequisite for starting (if it fails, the canary does not even begin); (b) during the canary, capture requests + responses from v2 in real time and run the judge asynchronously in a batch job that finishes before the next step; (c) use statistical embedding drift as a fast proxy for inline quality, and reserve the judge for intermediate gates between steps.

Pitfall 3 — SSE streaming complicates shadowing. Classic traffic mirroring (NGINX mirror, Istio MirrorPolicy) is designed for request/response HTTP: it copies the request, lets the primary server answer the client, and duplicates the request to the secondary, discarding the response. With SSE, the secondary’s response is a continuous stream of several seconds, and keeping two streams in parallel doubles the load on the proxy. Solutions: (a) shadow only non-streaming requests (chat without streaming, embeddings, classification, batch eval), (b) shadow the streaming traffic but with a short timeout on the secondary (discard the shadow if it takes longer than 30 s), (c) replace the shadow with a small-weight canary (1 %), which does handle streaming well.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB = 16 GPUs, serving Llama 70B FP8 with TP=4 (4 possible replicas, one per node):

Blue-green: impossible to keep two complete pools of 4 replicas without extra GPUs. Practical solution: blue-green with reduced pools (2 v1 replicas + 2 v2 replicas) during the window, with accepted capacity degradation (half the sustained RPS SLO), or having a parallel cluster (another node) reserved for rollouts.
Canary: feasible. Start with 3 v1 replicas + 1 v2 replica (25 % nominal weight, though the traffic weight also varies). Move to 2 v1 + 2 v2 at 50 %, then 1 v1 + 3 v2, finally 0 v1 + 4 v2.
Shadow: awkward because of the GPU cost. Reserve it for pre-canary validation of major changes, during a short window (4–8 hours) with shadowed traffic limited to a sample (10–20 % of requests, not 100 %).

For clusters of 8 GPU nodes, the three patterns are all sustainable. The operational rule: the rollout budget is typically 25–30 % of the cluster’s sustained capacity, and buying for the peak plus that head-room makes the capacity planning numbers add up.

What we have not covered (upcoming articles)

Multi-region rollouts: how to coordinate a canary when the cluster is geographically distributed.
A/B testing of prompts (not of models): the same model with two different system prompts, measuring conversion.
Embedding rollback: changing the embedding model of a RAG system means re-embedding the whole corpus, so the canary mechanics are different. See RAG corpus curation.
Feature flags for LLM: granularity per tenant or per feature within the same model.
End-to-end continuous deployment: integration with the retrain pipeline so a new adapter is promoted automatically after passing evals.

References

Argo Rollouts project — argoproj.io/argo-rollouts (Rollout and AnalysisTemplate CRDs).
Flagger project — fluxcd.io/flagger (alternative with automatic progression).
Istio — Traffic Mirroring (mirror configurable at VirtualService level).
NGINX Ingress — Canary annotations (nginx.ingress.kubernetes.io/canary-*).
vLLM project — issue tracker on hot model swap (status as of May 2026: in design, not production-ready).
Hou et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — reference on goodput metrics applicable to canary gates.
Bürkner et al. — Statistical methods for detecting model drift in production (various articles on Wasserstein and KL in ML monitoring).

LLM-as-judge: the exam marker who grades other models without turning into an oracle

Wed, 27 May 2026 08:30:00 +0200

This post goes deeper into the judges section of Evals for LLMs. There it was one piece of the mixed panel; here we get into why it works, where it breaks and how its calibration is measured before accepting it in CI.

TL;DR

An LLM judge is not “a GPT-4 you ask whether the answer is any good”. It is a trained marker: it has a rubric written in advance, it is asked for explicit reasoning before the verdict, its score is computed as an expectation weighted by token probabilities (not as the first token it spits out), and before being accepted in production it is calibrated against a sample of ~50 human-annotated examples until it reaches κ ≥ 0.5 (Cohen’s kappa). The state of the art in May 2026 is three patterns, G-Eval, Prometheus 2 and panel of judges, each answering a different trade-off between cost, quality and reproducibility. All of them share four documented biases: position, verbosity, self-preference and narcissism. This post explains how a real judge is built, how to measure whether it lies, and when each pattern is appropriate.

You are here: EVAL

The analogy: the civil service exam marker

A public competitive exam has thousands of papers. There is no way the senior board can mark them all. The Spanish system’s answer has been the same for decades: trained markers. People who are not professors, but who are given:

A marking template written in advance: what is valued, how many points each section is worth, what gets deducted.
Prior training with a sample of papers already marked by the senior board, until their marks agree reasonably well.
Continuous auditing: a fraction of their marking is re-marked by the senior board to verify that the marker is not drifting.

A well-built LLM judge is exactly that. It is not an oracle, it is a trained marker:

The rubric is the explicit criteria in the prompt (what faithfulness is, what relevancy is, and so on).
The training is the calibration against ~50 human-annotated examples.
The continuous auditing is the weekly sampling where a human re-evaluates a fraction of the traffic the judge has judged.

And, as with the human marker, the judge has systematic biases that the public exam system has learned to watch for: markers prefer certain handwriting styles, certain lengths, certain structures. The same happens to the judge. The rest of this post takes apart exactly which ones and how they are measured.

Why LLM-as-judge exists

The direct reason: money and time. A professional human annotation costs on the order of 0.50 € to 5 € per example (depending on complexity and domain), and takes 30 seconds to several minutes. A GPT-4 judgement costs ~0.01-0.05 € and takes ~2 seconds. For a golden dataset of 500 examples evaluated continuously over 10 candidates a day, the difference is between 2,500 € a day and 50 € a day. And the wall-clock difference is between days and minutes.

The less direct but more relevant reason: methodological scalability. A golden dataset of 500 examples is relatively easy to annotate once. What happens afterwards is the hard part:

Every time a candidate adapter appears, all 500 have to be re-evaluated.
Every time the dataset is updated (because a new incident came in) it has to be re-evaluated.
Every time the system prompt changes it has to be re-evaluated.

Without an automatable, cheap judge, the eval battery stops running and the system goes blind between releases. That is what really justifies the pattern.

The three canonical patterns in 2026

1 · G-Eval

Published by Liu et al. (2023). The base idea is so simple it fits in one sentence: give the judge a detailed rubric, ask it to reason before scoring, and read the result as a weighted expectation instead of as the first token. The three levers:

Rubric: a prompt with an explicit criterion (e.g. “Faithfulness: the degree to which the answer relies only on the context, without inventing data. Score 1 = invents everything, 5 = everything supported by the context”), ideally with one or two examples per extreme value.

Chain-of-thought + form-filling: the model is first asked to “reason briefly about each criterion” and then to “fill in this JSON form”. That forces it not to spit out an arbitrary number.

Probability-weighted scoring: instead of reading the first token after the score: field, you look at the model’s probability distribution over the tokens 1, 2, 3, 4, 5 and compute:

$$\hat{s} = \sum_{i=1}^{5} i \cdot \frac{p(\text{token}=i)}{\sum_{j=1}^{5} p(\text{token}=j)}$$

This turns a discrete score into a continuous one. The justification: if the judge was “torn” between 4 and 5 (probabilities 0.4 and 0.5 on 4 and 5), the real score is 4.55, not 5. This sharply reduces variance across runs of the same prompt, and captures information that greedy decoding throws away.

Limitations of G-Eval: it needs access to the model’s logprobs. Closed-source models have been restricting that access (Claude does not expose it, GPT-4 does but only top-5). In 2026 G-Eval with strict probability weighting is only practical against open source models you serve yourself (vLLM exposes them) or against GPT-4 with logprobs=true.

2 · Prometheus 2

Published by Kim et al. (KAIST, 2024). The insight is complementary to G-Eval: what if, instead of asking a generalist judge to evaluate, we fine-tune a specific judge?

Prometheus 2 is a Mistral 8×7B (MoE, ~47 GB in BF16, ~24 GB in INT4) fine-tuned on 100k+ evaluation examples with varied rubrics. The metric published in the paper: 0.897 Pearson correlation with GPT-4-as-judge on the Vicuna Bench and similar. That matters because it means GPT-4 can be replaced as a judge at the cost of local inference without losing almost any quality.

Why it matters in on-premise production:

No data leaves the perimeter. For strict ENS/NIS2 customers this is not a preference, it is a requirement. A judge that travels over an external API is not an option there.
Zero marginal cost. When the judge runs on-prem, evaluating 50,000 cases a day adds no external bill.
Controlled latency. Continuous eval over real traffic can run in parallel without saturating an external provider’s rate limits.

The price: you have to maintain one more inference service (Prometheus 2 running in its own vLLM), and the judge does not “update” unless it is re-fine-tuned.

3 · Panel of Judges

Verga et al. (Cohere, 2024) formalised it: instead of a single judge, use 3-5 heterogeneous judges, different models, different prompts, different temperatures, and aggregate their judgements.

Common aggregation mechanisms:

Median for continuous scores. Robust to outliers.
Majority vote for pairwise judgements (chosen vs rejected). If 3 of 5 prefer A, the winner is A.
Calibration-weighted mean: weight each judge by its κ against humans in calibration. The more reliable judges vote more.

What a panel gives that a single judge does not:

Reduction of self-preference (a judge’s bias towards outputs stylistically similar to its own): if the judges come from different vendors, the sum cancels out.
A measure of case difficulty: if all 5 judges agree, the case is easy; if they are split, the case is ambiguous and should be escalated to a human. That turns the panel into an automatic triaging system for human annotation.
Lower variance: each judge’s noise is averaged out.

The cost: 3-5× the bill of a single judge. That is why the panel is usually reserved for critical eval gates (does this adapter get promoted or not?), not for continuous eval over traffic.

How to measure whether the judge lies: Cohen’s kappa

Accepting the judge in production without measuring its agreement with humans is the same as accepting an uncalibrated thermometer. The standard metric for inter-rater agreement with discrete or ordinal scales is Cohen’s kappa:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Where p_o is the observed agreement proportion (what percentage of examples judge and human agree on) and p_e is the agreement proportion expected by chance (what you would expect if both scored at random respecting each one’s marginals).

The intuition: if p_o = 0.9 but p_e = 0.85 (because both nearly always score “4 or 5”), agreement is 90 % in raw terms but κ = 0.33: most of the agreement comes from both scoring high, not from them understanding each other. κ corrects for that baseline.

The usual interpretive scale (Landis and Koch 1977, still the reference):

κ	Interpretation	Production threshold
< 0.20	Poor	Useless
0.21–0.40	Fair	Weak signal only
0.41–0.60	Moderate	Minimum acceptable in 2026
0.61–0.80	Substantial	State of the art for open source judges
0.81–1.00	Almost perfect	Human judges rarely reach it between themselves

An important point the field learned the hard way: humans between themselves rarely exceed κ = 0.70 on LLM tasks (faithfulness, relevancy). That is the realistic ceiling for an LLM judge. Chasing κ = 0.9 against humans is chasing a ghost: not even two human annotators get there.

Weighted kappa for ordinal scales

For 1-5 scores, the disagreement “judge says 4, human says 5” is not the same as “judge says 1, human says 5”. Standard kappa treats both as an identical failure. Linear or quadratic weighted kappa assigns weight to the magnitude of the disagreement:

$$\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} \, o_{ij}}{\sum_{i,j} w_{ij} \, e_{ij}}, \quad w_{ij}^{(\text{lin})} = \frac{|i-j|}{k-1}, \quad w_{ij}^{(\text{quad})} = \frac{(i-j)^2}{(k-1)^2}.$$

In G-Eval with 1-5 scores, the usual practice is to publish κ_quad because it penalises large disagreements more and comes closer to human intuition.

Numerical calibration example

Imagine a golden set of 50 examples scored by a human and by the judge, both on a 1-5 scale. The confusion matrix:

	Hum 1	Hum 2	Hum 3	Hum 4	Hum 5	Judge total
Judge 1	3	1	0	0	0	4
Judge 2	1	4	2	0	0	7
Judge 3	0	1	6	2	0	9
Judge 4	0	0	1	12	3	16
Judge 5	0	0	0	2	12	14
Human total	4	6	9	16	15	50

Diagonal (exact agreements): 3+4+6+12+12 = 37 → p_o = 0.74.

p_e is computed as Σ_i (n_judge_i · n_hum_i) / n² = (4·4 + 7·6 + 9·9 + 16·16 + 14·15) / 50² = (16+42+81+256+210) / 2500 = 605/2500 ≈ 0.242.

$$\kappa = \frac{0.74 - 0.242}{1 - 0.242} = \frac{0.498}{0.758} \approx 0.66$$

Substantial. Acceptable. If we wanted quadratic weighted κ, near disagreements (Judge 4 vs Hum 5) weigh less than distant ones (Judge 2 vs Hum 4), and κ_quad typically comes out 0.05-0.10 above the linear one.

The four biases of the judge

Position bias

First documented by Wang et al. (2023). If you present two answers A and B to the judge in a pairwise setting, it prefers A more often than B, on the order of 55-65 % when A and B are objectively equivalent, with typical 2023-2024 judges. In 2026 frontier judges (GPT-5, Claude 4.5, Llama 4 Judge) have it fairly well mitigated, but it is still measurable.

How to measure it formally: run the dataset twice, once with (A, B) and once with (B, A). If the judge is consistent, the two passes should agree. The fraction of cases where the verdict flips is the position-bias rate. Accepted threshold in 2026: < 10 %.

Canonical mitigation: always run two passes with the order reversed and average. Frameworks such as Promptfoo and Inspect AI do it by default.

Verbosity bias

Documented by Saito et al. and Dubois et al. in 2024. For open-ended tasks (faithfulness, helpfulness), the judge tends to give a higher score to longer answers. The typical Pearson correlation between score and length on answers of equivalent human-rated quality can rise to 0.4-0.5 without mitigation.

How to measure it: Pearson correlation between the judge’s score and the length of the answer, computed over a subset where humans have confirmed equivalent quality. If the correlation is > 0.3 on that controlled subset, there is significant verbosity bias.

Mitigation: a rubric explicitly neutral on length (“the answer should be appropriately concise for the question; length is not a criterion”) and few-shot examples where a short answer beats a long one. AlpacaEval 2.0 builds a length correction directly into the metric.

Self-preference bias

Documented by Panickssery et al. (Anthropic, 2024). A GPT-4 judge prefers GPT-4 outputs. A Claude judge prefers Claude outputs. Not out of conspiracy, but because models share stylistic patterns with their close relatives (paragraph structure, use of bullet points, tone).

How to measure it: over a golden set, compare the scores of 3 different judges (e.g. GPT-4, Claude, Llama 4 Judge). If for the same candidate there is > 15 % systematic divergence attributable to the candidate’s identity, there is self-preference.

Mitigation: a panel of judges with heterogeneous vendors. If a single judge is going to be used, it must not come from the same vendor as the model being evaluated.

Narcissism

The extreme case of self-preference: the judge is exactly the same model as the candidate. This happens more than it seems: a team trains a Llama 3 8B with LoRA and evaluates it with Llama 3 8B as the judge because “it is what they have on-prem”. It is methodologically invalid. The delta between human score and judge score grows measurably.

Mitigation: a judge of a different architecture from the candidate. If your candidate is Llama 3, your judge should be Mistral, Qwen or a Prometheus 2 (which, although based on Mistral, was fine-tuned specifically for evaluation).

The judge as the pipeline’s hinge

The judge is not just “the eval piece”. It is the hinge that connects three stages of the pipeline:

Towards TUNE: the judge generates the (chosen, rejected) pairs for DPO without needing human labellers. That chain is what makes continuous fine-tuning work without a dedicated annotation team.

Towards the EVAL gate: the judge gives the score that is compared against the CI threshold. If the adapter does not clear 0.85 on faithfulness, no merge.

Towards continuous EVAL: over a sample of real traffic (1-5 %), the judge computes scores and persists them. That makes it possible to detect regressions that appear days after deployment and that CI did not see because its golden set did not cover them.

Towards RETRAIN: the cases where the judge gives a low score are automatic candidates for the next retraining dataset. The judge acts as the triage of the incident flow.

Implications for on-premise hardware

The numbers below are indicative for typical scenarios in May 2026.

On an RTX 4090 (24 GB)

Judge	Does it fit?	Approximate throughput	Notes
GPT-4o (API)	n/a	~50-100 judgements/min	External cost, not local
Prometheus 2 (8×7B INT4)	Yes, just about	~40-80 judgements/min	Q4_K_M GGUF, llama.cpp
Llama 3.1 8B fine-tuned judge	Yes, comfortably	~150-250 judgements/min	Reasonable on-prem default
Mistral Small Judge 22B	Not directly, requires offload	~10-20 judgements/min	Too much for 24 GB in BF16

Conclusion: on a single 4090, an open source 8B judge fine-tuned for evaluation (or a quantised Prometheus 2) is the sweet spot.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Judge	Configuration	Approximate throughput	Notes
Prometheus 2 BF16	TP=2	~400-700 judgements/min	Fits comfortably, low latency
Llama 3.3 70B Instruct	TP=4	~150-300 judgements/min	If used as a generalist judge
Panel of 3 judges in parallel	TP=1-2 each	~600-1200 judgements/min combined	Natural pattern in a cluster

On an NVLink cluster the natural approach is to run a panel of judges in parallel (each judge taking 1-2 GPUs) with a LiteLLM router in front. That removes the cognitive cost of “which judge do we use” because all three are used and the result is aggregated.

What we have not covered (upcoming articles)

Red teaming and safety eval: how robustness against adversarial prompts is evaluated. It is a different pattern from the ordinal judge.
Multistep agent eval: AgentBench, TauBench, evaluating trajectories instead of individual outputs.
Benchmark contamination: how to detect whether the evaluated model saw the golden set during pre-training, and why public benchmarks are half broken.
Cost-aware judging: when a cheap judge (Llama 8B) beats an expensive one (GPT-4o), and how to quantify the quality/cost trade-off with Pareto curves.

References

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (EMNLP 2023).
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., Seo, M. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (EMNLP 2024).
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Cohere, 2024).
Panickssery, A., Bowman, S., Feng, S. LLM Evaluators Recognize and Favor Their Own Generations (Anthropic, NeurIPS 2024).
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., Sui, Z. Large Language Models are Not Fair Evaluators (ACL 2024).
Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Stanford, 2024).
Cohen, J. A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement, 1960).
Landis, J. R., Koch, G. G. The Measurement of Observer Agreement for Categorical Data (Biometrics, 1977).

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

Mon, 25 May 2026 07:30:00 +0200

This is stage 3 of the LLMOps pipeline. If you arrive without context on the full journey, the six-stage LLMOps pipeline describes where Eval sits between Tune and Deploy, and the anatomy of a production LLM request shows a real eval blocking the promotion of an adapter.

TL;DR

Tracing is not evaluation. Tracing tells you what happened; eval tells you whether what happened is any good. The two layers live in tools that overlap visually (Langfuse does both), but the discipline is different: tracing is continuous capture over real traffic; eval is controlled execution against a stable dataset, with metrics that have to fail CI if they drop below a threshold. Without eval, the Tune → Deploy loop closes blind: adapter v8 goes to production because the engineer “saw it answering well” on five examples. With eval done properly, v8 only gets in if it beats v7 across a battery of 500 curated cases, evaluated by a mix of heuristics, embeddings, an LLM judge calibrated against humans, and a sample of real traffic with human annotation. This post takes apart the mechanism, the mathematics that keep you honest, the real tools in 2026 and the pitfalls that turn it into theatre.

The analogy: the academic examination board

A doctoral candidate defends a thesis. They do not defend it before a single distracted professor: they defend it before a mixed board, with a syllabus fixed in advance (not improvised on the day), and with an explicit pass mark that separates pass from fail. The board is not a single expert: it is a panel combining fast readers (the heuristics: does it have the requested format? does it include the citation?), semantic reviewers (the embeddings: does it resemble the expected answer?), an external assessor trained on the subject (the LLM judge: is it faithful to the context, is it relevant, does it sound coherent?), and human jury members on a sample of the most sensitive cases. If the candidate does not reach the mark, they are not promoted: they go back and prepare the defence again.

That analogy has three edges worth holding on to from the first minute:

The syllabus is fixed, but it is actively updated when a new topic appears in the real world that the candidate must know. If it is not updated, the exam measures something increasingly distant from what happens outside.
The board has to be trained: an LLM judge not calibrated against humans is an examiner making up the criteria.
The pass mark is published beforehand: it is not decided after seeing the result, because then it is not a pass mark, it is a justification.

These three ideas run through the rest of the post. Every tool and every piece of mathematics that follows is, at bottom, a way of operationalising them.

The mechanism itself: four layers of evaluators

An eval suite in 2026 is made up of four layers that coexist. None replaces the others; each one measures what the rest cannot measure well, and leaves unmeasured what the rest do measure well.

Layer 1 — Deterministic heuristics. Rules that return true or false with no ambiguity: the output matches a regex, contains a specific entity, does not exceed a length, follows a valid JSON schema, respects a requested format (markdown, tool_call function, mandatory citation). They are cheap, extremely fast, need neither judge nor embeddings, and catch the most frequent class of bug: the model answered in the wrong format. Their limit is obvious, in that they do not know whether the answer is correct, only whether it is well formed.

Layer 2 — Semantic metrics with embeddings. They compare the model’s output with an expected answer by computing cosine similarity between their embeddings, or by measuring whether a claim in the output is entailed by the retrieved context. They are cheap, deterministic given the embedding model, and very useful for detecting answers that drift in meaning but not in form. Their limit is equally clear: two answers can have high cosine similarity and say the opposite of each other (“the customer can cancel at any time” vs “the customer cannot cancel at any time” share 80% of their tokens).

Layer 3 — LLM-as-judge. A model, or an ensemble, evaluates the output of the model under test with a prompt designed to produce a score on a rubric. The canonical methods in 2026 are G-Eval (chain-of-thought prompting with a calibrated numeric score), Prometheus (open-source judge trained specifically for evals, reporting 0.897 correlation with humans in its v2.5 release from late 2025), and panel-of-judges setups that average votes from three heterogeneous models to reduce bias. This layer captures nuances the previous two cannot see: is it faithful to the context? is it useful? is it safe? is it complete? Its limit is cost and the need for calibration, treated as a whole section below.

Layer 4 — Humans. Trained annotators who evaluate a sample of the eval set, not all of it. They are the only layer with ultimate authority over the rubric: the LLM judge is calibrated against them, not the other way round. They are expensive (≈ 0.50–2.00 € per annotated sample when the domain is technical) and slow (a competent annotator produces 60–120 quality annotations per working day). The mistake teams make again and again is dropping this layer “because we have an LLM judge”; without humans there is no calibration, and without calibration the judge measures whatever it feels like.

Normal operation of an eval gate combines all four: heuristics remove malformed outputs before spending judge tokens, embeddings filter out the manifestly irrelevant, the judge scores the rest, and humans annotate a sample every N runs to keep the judge calibrated.

The golden dataset: a versioned syllabus

The golden dataset is the most undervalued artefact in the pipeline. It is the exam. If it is badly built, everything else, the best-calibrated judge in the world, the strictest gates, the fastest suite, measures noise. Covered at the top level in the data versioning post as one of the four artefacts to version differently; here we go into detail from the Eval perspective.

The three properties a golden dataset has to satisfy are:

Stratified representativeness. The dataset has to cover the system’s real input space in proportions that reflect production. If 30% of real traffic is in German, 30% of the golden set has to be in German; if 12% of questions are about subscription cancellation, that category cannot be 60% of the eval set just because it was easy to annotate. Stratification stays auditable: each example carries tags (lang=de, category=cancellation, tenant_type=enterprise, difficulty=hard) and the suite reports metrics per segment, not just the aggregate.

Strict holdout, no contamination with training. This rule is so obvious that almost every team believes it is following it, and almost every team breaks it without noticing. If the golden eval set gets mixed with the fine-tuning dataset, because somebody did a badly built random_split, because a purchased dataset is used by other vendors for training, because the LLM judge saw it during pretraining, then the metric stops measuring generalisation and starts measuring memorisation. The eval set hash is versioned separately (covered in the continuous fine-tuning post) and a routine leakage check is run: if an example in the golden set matches one in training token for token, it raises an alert.

Reasoned sample size, not aspirational. How many examples are needed? There is a mathematical minimum for telling two models apart with confidence. If model A is right 80% of the time and model B 85% on the same set, the 95% confidence interval for that difference of proportions (unpaired) is:

\[ \Delta p \pm 1{.}96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

To distinguish 80% from 85% with 95% confidence (an interval that does not cross zero), you need n ≈ 700 examples. To distinguish 90% from 91% under the same criterion, the calculation gives n ≈ 6,500. A “we have 50 examples in the golden set and we see adapter v8 scoring 90%” means nothing statistically: the confidence interval is ±8 points. The field’s practical rule in 2026 is a minimum of 300 examples to detect coarse differences, ideally 500–1,500 if you want to detect fine improvements, and to start by stratifying properly before obsessing over sample size.

On top of this comes active maintenance: the golden set is enriched with production incidents (covered in detail in the retrain post), so that every real complaint that ends in a bug becomes a curated example the next deploy candidate will have to pass. The golden set is not static: it is a living record of errors the system has already made and must not make again.

LLM-as-judge: how you calibrate an examiner

Layer 3 is the one most teams misuse. The typical mistake is: “we use GPT-4 as judge because it is the most capable”. The judge is not chosen for nominal capability; it is chosen for agreement with humans on the specific rubric you are measuring. A judge with 60% agreement is useless even if it is GPT-5; a Prometheus 7B fine-tuned for your domain with 88% agreement is worth more.

The standard metric for measuring agreement between two annotators (human vs judge, or two humans with each other) is Cohen’s kappa, which corrects for the agreement expected by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]

where $p_o$ is the observed proportion of agreement and $p_e$ is the proportion expected by chance under each annotator’s marginal distributions. The accepted interpretations in the literature are:

κ < 0.40: poor agreement. The judge says whatever it likes.
κ ∈ [0.40, 0.60]: moderate. Acceptable for coarse signals (is it toxic?), dreadful for nuance (is it faithful to the context?).
κ ∈ [0.60, 0.80]: substantial. Useful in production for most metrics.
κ > 0.80: almost perfect. The judge can be treated as a substitute for the human for that specific kind of judgement.

Numerically, consider a binary rubric (faithful / not faithful) over 200 examples annotated by a human and by a judge. If the human said “faithful” in 150 cases and the judge in 140, and they agree on 175 of the 200, then $p_o = 0{.}875$; the marginals are $p_h = 0{.}75$, $p_j = 0{.}70$, and $p_e = 0{.}75 \cdot 0{.}70 + 0{.}25 \cdot 0{.}30 = 0{.}600$. Kappa comes out at 0.6875: substantial but not excellent, usable, with an eye kept on the rubric.

Calibrating the judge means an explicit process:

Build a calibration set — 100–300 examples annotated by trained humans, with written guidelines. Inter-annotator kappa between the humans is also measured; if the humans themselves cannot agree, the rubric is badly written before you even start talking about a judge.
Iterate the judge prompt until judge agreement with the humans passes the accepted threshold (typically κ ≥ 0.7 for sensitive metrics).
Pin the judge version (claude-3-5-sonnet-20251022, gpt-4o-2024-11, prometheus-2-7b@sha256:…): any change invalidates the calibration.
Recalibrate periodically — every time the judge, the judge prompt, or the rubric changes. The frequency the field recommends in 2026 is quarterly at minimum, monthly if the rubric is new.
Persist everything in lineage — a score of “faithfulness 0.87” with no traceability of which judge, which prompt, which calibration set and which human validated it, is decorative.

The prompt versioning post covers how judge prompt versioning is materialised. The practical consequence is that the judge is versioned like any other model: your eval suite has judge_id = prometheus-2.5@v3 just as it has adapter_id = customer_support_v7.

The two cadences: CI gate and platform regression

Eval suites live in two places and run on two different cadences. Teams that conflate them turn one of the two into theatre.

CI gate (pre-merge, blocking). It runs on every pull request that modifies prompts, adapters, RAG configuration, or any artefact that could move the model’s output. It runs against the golden dataset versioned at the hash that is on main. The gate fails the merge if:

the critical metric drops more than X percentage points in absolute terms against the baseline (typically X = 2);
some safety metric (toxicity, PII leakage) crosses a hard threshold (typically tox > 0.02);
some strategic segment (a language, an enterprise-type tenant) drops more than Y points even if the aggregate improves.

This cadence has to be fast (ideally < 10 minutes over 500 examples) and cheap (LLM judge in batch mode, cached embeddings, heuristics run locally). The CI gate is not exhaustive: it is the low-latency line of defence.

Platform regression (post-deploy, continuous). It runs on a schedule (typically nightly or weekly) over sampled production traffic, not over the static golden set. It detects drift: the model has not changed, the golden set has not changed, but the users have, and quality over real traffic drops. This cadence is more expensive (judge over thousands of samples, human annotation over hundreds), tolerates latencies of hours, and its main consumer is not CI but the observability dashboard and the humans on the product team who decide whether to open a retrain cycle.

Both cadences persist results in the same store (Langfuse, MLflow, or equivalent) and connect them by model_id, prompt_id, dataset_hash and judge_id. Without that glue of identifiers, the metric that passes CI cannot be correlated with the one that fails in production three weeks later.

The minimum mathematics that matter

Beyond kappa and the interval for the difference of proportions, both already covered above, there are three more mathematical pieces that any team running evals seriously ends up using.

Confidence interval for a continuous metric. If your metric is a continuous score (faithfulness ∈ [0, 1]) and you measure the sample mean $\bar{x}$ over n examples with standard deviation s, the 95% confidence interval for the population mean is:

\[ \bar{x} \pm 1{.}96 \cdot \frac{s}{\sqrt{n}} \]

For n = 300 and s ≈ 0.2 (typical of a 0-1 score with non-degenerate variance), the margin is ±0.023. This means that differences below 2 hundredths cannot be distinguished from noise at that sample size. If your team is chasing “+0.5 pp” improvements over 100 examples, it is optimising noise.

Judge cost as a function of sample size and rubric. The cost of one eval pass with LLM-as-judge over n examples, with m metrics evaluated in a single call per example, and per-token price $c_{in}, c_{out}$ on the judge model, is:

\[ C \approx n \cdot (t_{in} \cdot c_{in} + t_{out} \cdot c_{out}) \]

where $t_{in}$ is the number of input tokens (includes context, output of the model under test, full rubric) and $t_{out}$ the output tokens (includes the judge’s CoT + score). For n = 500, $t_{in}$ ≈ 4,000, $t_{out}$ ≈ 300, a GPT-4o judge at May 2026 prices, one pass costs on the order of 8–15 USD per suite. If the pass fires on every PR and there are 30 PRs/day, that is 240–450 USD/day on CI gates alone. Multiplied by continuous regression, teams that do not control this spend four figures a month on judge tokens without realising. The canonical mitigation is a mix of layers: heuristics and embeddings filter first, the judge is only invoked on what the cheap layers cannot resolve, and for platform regression a self-hosted open-source judge is used (Prometheus 7B on your own GPU plane) instead of a commercial model.

Distinction between aggregate metric and per-segment metric. The classic eval fallacy is the hidden mean. If your suite reports faithfulness = 0.87 and the team reads it as “up 2 points against the previous adapter”, this may be what is happening: the new adapter gains 4 points in English (where 70% of the eval set is) and loses 6 points in German (where 30% is). The aggregate mean improves, the German experience gets worse. Any serious suite reports a breakdown by strategic segment (language, tenant type, question category, context length). The CI gate can also have per-segment thresholds, not just aggregate ones.

The 2026 stack: dominant tools

Tool	Main layer	Licence	Maintainer	When to pick it
DeepEval	CI gate	Apache 2.0	Confident AI	“Evals as pytest” — assertions in Python code, trivial integration with GitHub Actions. A reasonable default.
Promptfoo	CI gate	MIT	Promptfoo Inc.	Declarative YAML, matrix of prompts × providers × assertions, diff vs baseline. DevOps-friendly.
RAGAS	RAG-specific metrics	Apache 2.0	Exploding Gradients	Faithfulness, context relevancy, answer relevancy. The canonical piece if your system is RAG.
Inspect AI	Safety/capability evals	MIT	UK AI Safety Institute	Suite focused on safety and capability. Useful for regulatory gates under the EU AI Act.
Langfuse Evals	Platform regression	MIT (OSS) / EE	Langfuse GmbH	Integrated with tracing — datasets, runs and scores in the same UI as production traces.
MLflow GenAI Evals	Registry + evals	Apache 2.0	Databricks/LF AI	Good when you already have MLflow for classic models; “GenAI dashboard” since 3.10.
Phoenix (Arize)	Eval + visual drift	Elastic License 2.0	Arize AI	Focused on visual debugging of embeddings and drift; a complement to Langfuse, not a replacement.
Prometheus 2.5	Self-hosted OSS judge	Apache 2.0	KAIST + LG AI	Fine-tuned judge, high correlation with GPT-4 at zero cost per token when self-hosted.
G-Eval / JudgeLM	Prompting methods	— (techniques)	academic	Prompting frameworks for LLM-as-judge — they apply on top of any judge model.

The OSS catalogue by stage goes into detail entry by entry; the parallel OSS vs hyperscalers catalogue compares against Bedrock Evaluations, Vertex AI Eval Service and Azure AI Evaluation.

The canonical pattern in 2026 is hybrid: DeepEval or Promptfoo for CI gates + Langfuse Evals for platform regression + Prometheus 2.5 as a self-hosted judge + human annotation on Argilla or Label Studio for the calibration set. Swapping any of these pillars for equivalents (W&B Weave instead of Langfuse, Inspect instead of DeepEval) is style, not functionality; what matters is that the four functions are present and connected.

Applied to typical on-premise hardware

For an on-premise deployment that wants to avoid sending sensitive outputs to a commercial LLM judge (for data sovereignty, ENS, NIS2 or equivalents), the judge is hosted on your own GPU plane. Reference figures for May 2026, based on Prometheus 2.5 (Llama-3.1-8B fine-tuned as a judge) served on vLLM:

RTX 4090 (24 GB, Ada Lovelace): viable for small suites (< 200 examples) and for running the judge offline during development. Latency per evaluation ≈ 1.5–3 s with BF16; aggregate throughput on the order of 25–40 evaluations/min with batching. Useful for a developer’s local CI gates, not for platform regression.
Generic 4×H100 SXM configuration (320 GB total, NVLink): runs the judge in parallel at TP=2 across two GPUs, leaving two free to serve the model under test. Aggregate throughput on the order of 200–350 evaluations/min, a full 500-example suite in 2–3 min. This allows PR gates with no perceptible waiting and nightly platform regression over thousands of samples with no per-token cost.

The comparative cost arithmetic is stubborn: self-hosting Prometheus 2.5 pays off an H100 in roughly 6 months if the team fires ≥ 30 PRs/day with eval gates. Below that volume, the commercial judge still wins unless sovereignty is a requirement, and under ENS / NIS2 it is.

The seven pitfalls that kill this stage

Pitfall 1 — Aged golden dataset. It is not enriched with production incidents. After months it measures a world that no longer exists. The metric climbs comfortably while real users complain more.

Pitfall 2 — Contaminated or uncalibrated judge. The LLM judge evaluates with criteria it invents itself. Without a human calibration set as reference, there is no way to know whether its 0.89 is generous, severe or random.

Pitfall 3 — Insufficient sample size. A 50-example suite, 1-point differences the team treats as significant. The confidence interval is ±10 points. They are measuring noise and making real decisions on it.

Pitfall 4 — Runaway cost. A GPT-4 batch-mode judge fired on every PR over 1,000 examples, with no prior filtering by cheap layers. The eval bill overtakes production serving. It happens more often than people admit.

Pitfall 5 — Aggregate metric hiding segments. Global mean improves 2 points, German drops 6, enterprise tenants drop 3. With no explicit per-segment breakdown, the gate approves what it should not.

Pitfall 6 — Judge with a floating version. Judge model updated without recalibrating the rubric. Thresholds lose their statistical meaning. Last month’s regressions are not comparable with those from two months ago.

Pitfall 7 — Eval gate that is not enforced. The gate exists in the documentation but not in the real workflow: the suite takes 30 minutes, developers skip it with --no-verify, managers ask for one-off exceptions that become the norm. An eval gate without enforcement is ornament.

All seven are operational, not technical. The Eval layer does not break because the mathematics are wrong: it breaks because the discipline slackens. It is the same thing that happens with unit tests in any project that grows, except that here, without the discipline, the system improves its metrics while getting worse, and that makes the degradation invisible until it is already ungovernable.

What we have not covered (upcoming posts)

Guardrails and online safety: the inline eval layer that filters outputs in real time, not in CI. Conceptually a cousin of Eval, but with very different latency constraints. Covered in detail in the guardrails post: four lines of defence (input, retrieval, tool, output), OWASP LLM Top 10 (2025), OSS catalogue and deployment patterns.
Judge ensembles and agreement between multiple judges: how to reduce bias by combining three heterogeneous models as a panel, which aggregation function works (simple majority, trimmed mean, calibrated meta-judge).
Meta-eval: how you evaluate the suite itself. If the eval improvement from adapter v7 to v8 does not translate into a real improvement for the user, the suite is wrong, and that is also measured, by correlating eval metrics against product metrics.
Metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, alternative language) as an extra gate. Beyond nominal agreement, it measures consistency.
Adversarial evals with red teaming: introducing prompt injection and jailbreak attacks as part of the gate.
Privacy in the judge: how to stop sensitive outputs from the model under test travelling to an external judge when regulation forbids it — homomorphic judge, judge in a TEE, or simply a self-hosted judge.

References

G-Eval: Liu et al., “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” — the reference paper for the chain-of-thought LLM judge method.
Prometheus: Kim et al., “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models” — OSS judge with a reported 0.897 correlation vs humans.
RAGAS: Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation” — the paper that standardises faithfulness, context relevancy and answer relevancy.
Cohen’s kappa: Cohen, J. (1960). “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement — the classic metric for inter-annotator agreement, still the operational reference.
DeepEval docs: https://docs.confident-ai.com/
Promptfoo docs: https://promptfoo.dev/docs/
Langfuse Evals: https://langfuse.com/docs/scores
Inspect AI: https://inspect.ai-safety-institute.org.uk/
EU AI Act, relevant articles on mandatory evaluation of high-risk systems — pending publication of the CEN/CENELEC technical standards on conformity assessment for GenAI.

Llm-as-Judge on lo0 — Tech Blog

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

TL;DR

The master analogy: the quality inspector at a furniture factory

The problem of evaluating RAG

The four RAGAS metrics

1. Faithfulness — fidelity to the context

2. Answer Relevance — relevance of the answer

3. Context Precision — retrieval precision

4. Context Recall — retrieval recall

5. Noise Sensitivity — sensitivity to noise

A complete worked example

Scenario

Step-by-step calculation

Summary of the example

Building the golden dataset

What it is and why it matters

LLM-assisted construction pipeline

Minimum size

Stratifying the dataset

The Goodhart trap

Correlation with real satisfaction

Diagram: the continuous evaluation loop

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

Langfuse for eval traceability

Prometheus + Grafana for operational alerts

Running RAGAS against on-premise vLLM — practical considerations

What we have not covered

See also

References

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

TL;DR

You are here: DEPLOY (and the transition to RETRAIN)

The analogy: opening night of a play

The three strategies in detail

Blue-green

Canary

Shadow / mirror

The five regression metrics that act as a gate

The mechanics on Kubernetes: Argo Rollouts

The vLLM detail: why there is no “hot swap” of the model

The three pitfalls specific to LLM rollout

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References

LLM-as-judge: the exam marker who grades other models without turning into an oracle

TL;DR

You are here: EVAL

The analogy: the civil service exam marker

Why LLM-as-judge exists

The three canonical patterns in 2026

1 · G-Eval

2 · Prometheus 2

3 · Panel of Judges

How to measure whether the judge lies: Cohen’s kappa

Weighted kappa for ordinal scales

Numerical calibration example

The four biases of the judge

Position bias

Verbosity bias

Self-preference bias

Narcissism

The judge as the pipeline’s hinge

Implications for on-premise hardware

On an RTX 4090 (24 GB)

On a generic 4×H100 SXM cluster (320 GB, NVLink)

What we have not covered (upcoming articles)

See also

References

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

TL;DR

The analogy: the academic examination board

The mechanism itself: four layers of evaluators

The golden dataset: a versioned syllabus

LLM-as-judge: how you calibrate an examiner

The two cadences: CI gate and platform regression

The minimum mathematics that matter

The 2026 stack: dominant tools