Evals on lo0 — Tech Blog

LLM-as-judge: the exam marker who grades other models without turning into an oracle

Wed, 27 May 2026 08:30:00 +0200

This post goes deeper into the judges section of Evals for LLMs. There it was one piece of the mixed panel; here we get into why it works, where it breaks and how its calibration is measured before accepting it in CI.

TL;DR

An LLM judge is not “a GPT-4 you ask whether the answer is any good”. It is a trained marker: it has a rubric written in advance, it is asked for explicit reasoning before the verdict, its score is computed as an expectation weighted by token probabilities (not as the first token it spits out), and before being accepted in production it is calibrated against a sample of ~50 human-annotated examples until it reaches κ ≥ 0.5 (Cohen’s kappa). The state of the art in May 2026 is three patterns, G-Eval, Prometheus 2 and panel of judges, each answering a different trade-off between cost, quality and reproducibility. All of them share four documented biases: position, verbosity, self-preference and narcissism. This post explains how a real judge is built, how to measure whether it lies, and when each pattern is appropriate.

You are here: EVAL

The analogy: the civil service exam marker

A public competitive exam has thousands of papers. There is no way the senior board can mark them all. The Spanish system’s answer has been the same for decades: trained markers. People who are not professors, but who are given:

A marking template written in advance: what is valued, how many points each section is worth, what gets deducted.
Prior training with a sample of papers already marked by the senior board, until their marks agree reasonably well.
Continuous auditing: a fraction of their marking is re-marked by the senior board to verify that the marker is not drifting.

A well-built LLM judge is exactly that. It is not an oracle, it is a trained marker:

The rubric is the explicit criteria in the prompt (what faithfulness is, what relevancy is, and so on).
The training is the calibration against ~50 human-annotated examples.
The continuous auditing is the weekly sampling where a human re-evaluates a fraction of the traffic the judge has judged.

And, as with the human marker, the judge has systematic biases that the public exam system has learned to watch for: markers prefer certain handwriting styles, certain lengths, certain structures. The same happens to the judge. The rest of this post takes apart exactly which ones and how they are measured.

Why LLM-as-judge exists

The direct reason: money and time. A professional human annotation costs on the order of 0.50 € to 5 € per example (depending on complexity and domain), and takes 30 seconds to several minutes. A GPT-4 judgement costs ~0.01-0.05 € and takes ~2 seconds. For a golden dataset of 500 examples evaluated continuously over 10 candidates a day, the difference is between 2,500 € a day and 50 € a day. And the wall-clock difference is between days and minutes.

The less direct but more relevant reason: methodological scalability. A golden dataset of 500 examples is relatively easy to annotate once. What happens afterwards is the hard part:

Every time a candidate adapter appears, all 500 have to be re-evaluated.
Every time the dataset is updated (because a new incident came in) it has to be re-evaluated.
Every time the system prompt changes it has to be re-evaluated.

Without an automatable, cheap judge, the eval battery stops running and the system goes blind between releases. That is what really justifies the pattern.

The three canonical patterns in 2026

1 · G-Eval

Published by Liu et al. (2023). The base idea is so simple it fits in one sentence: give the judge a detailed rubric, ask it to reason before scoring, and read the result as a weighted expectation instead of as the first token. The three levers:

Rubric: a prompt with an explicit criterion (e.g. “Faithfulness: the degree to which the answer relies only on the context, without inventing data. Score 1 = invents everything, 5 = everything supported by the context”), ideally with one or two examples per extreme value.

Chain-of-thought + form-filling: the model is first asked to “reason briefly about each criterion” and then to “fill in this JSON form”. That forces it not to spit out an arbitrary number.

Probability-weighted scoring: instead of reading the first token after the score: field, you look at the model’s probability distribution over the tokens 1, 2, 3, 4, 5 and compute:

$$\hat{s} = \sum_{i=1}^{5} i \cdot \frac{p(\text{token}=i)}{\sum_{j=1}^{5} p(\text{token}=j)}$$

This turns a discrete score into a continuous one. The justification: if the judge was “torn” between 4 and 5 (probabilities 0.4 and 0.5 on 4 and 5), the real score is 4.55, not 5. This sharply reduces variance across runs of the same prompt, and captures information that greedy decoding throws away.

Limitations of G-Eval: it needs access to the model’s logprobs. Closed-source models have been restricting that access (Claude does not expose it, GPT-4 does but only top-5). In 2026 G-Eval with strict probability weighting is only practical against open source models you serve yourself (vLLM exposes them) or against GPT-4 with logprobs=true.

2 · Prometheus 2

Published by Kim et al. (KAIST, 2024). The insight is complementary to G-Eval: what if, instead of asking a generalist judge to evaluate, we fine-tune a specific judge?

Prometheus 2 is a Mistral 8×7B (MoE, ~47 GB in BF16, ~24 GB in INT4) fine-tuned on 100k+ evaluation examples with varied rubrics. The metric published in the paper: 0.897 Pearson correlation with GPT-4-as-judge on the Vicuna Bench and similar. That matters because it means GPT-4 can be replaced as a judge at the cost of local inference without losing almost any quality.

Why it matters in on-premise production:

No data leaves the perimeter. For strict ENS/NIS2 customers this is not a preference, it is a requirement. A judge that travels over an external API is not an option there.
Zero marginal cost. When the judge runs on-prem, evaluating 50,000 cases a day adds no external bill.
Controlled latency. Continuous eval over real traffic can run in parallel without saturating an external provider’s rate limits.

The price: you have to maintain one more inference service (Prometheus 2 running in its own vLLM), and the judge does not “update” unless it is re-fine-tuned.

3 · Panel of Judges

Verga et al. (Cohere, 2024) formalised it: instead of a single judge, use 3-5 heterogeneous judges, different models, different prompts, different temperatures, and aggregate their judgements.

Common aggregation mechanisms:

Median for continuous scores. Robust to outliers.
Majority vote for pairwise judgements (chosen vs rejected). If 3 of 5 prefer A, the winner is A.
Calibration-weighted mean: weight each judge by its κ against humans in calibration. The more reliable judges vote more.

What a panel gives that a single judge does not:

Reduction of self-preference (a judge’s bias towards outputs stylistically similar to its own): if the judges come from different vendors, the sum cancels out.
A measure of case difficulty: if all 5 judges agree, the case is easy; if they are split, the case is ambiguous and should be escalated to a human. That turns the panel into an automatic triaging system for human annotation.
Lower variance: each judge’s noise is averaged out.

The cost: 3-5× the bill of a single judge. That is why the panel is usually reserved for critical eval gates (does this adapter get promoted or not?), not for continuous eval over traffic.

How to measure whether the judge lies: Cohen’s kappa

Accepting the judge in production without measuring its agreement with humans is the same as accepting an uncalibrated thermometer. The standard metric for inter-rater agreement with discrete or ordinal scales is Cohen’s kappa:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Where p_o is the observed agreement proportion (what percentage of examples judge and human agree on) and p_e is the agreement proportion expected by chance (what you would expect if both scored at random respecting each one’s marginals).

The intuition: if p_o = 0.9 but p_e = 0.85 (because both nearly always score “4 or 5”), agreement is 90 % in raw terms but κ = 0.33: most of the agreement comes from both scoring high, not from them understanding each other. κ corrects for that baseline.

The usual interpretive scale (Landis and Koch 1977, still the reference):

κ	Interpretation	Production threshold
< 0.20	Poor	Useless
0.21–0.40	Fair	Weak signal only
0.41–0.60	Moderate	Minimum acceptable in 2026
0.61–0.80	Substantial	State of the art for open source judges
0.81–1.00	Almost perfect	Human judges rarely reach it between themselves

An important point the field learned the hard way: humans between themselves rarely exceed κ = 0.70 on LLM tasks (faithfulness, relevancy). That is the realistic ceiling for an LLM judge. Chasing κ = 0.9 against humans is chasing a ghost: not even two human annotators get there.

Weighted kappa for ordinal scales

For 1-5 scores, the disagreement “judge says 4, human says 5” is not the same as “judge says 1, human says 5”. Standard kappa treats both as an identical failure. Linear or quadratic weighted kappa assigns weight to the magnitude of the disagreement:

$$\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} \, o_{ij}}{\sum_{i,j} w_{ij} \, e_{ij}}, \quad w_{ij}^{(\text{lin})} = \frac{|i-j|}{k-1}, \quad w_{ij}^{(\text{quad})} = \frac{(i-j)^2}{(k-1)^2}.$$

In G-Eval with 1-5 scores, the usual practice is to publish κ_quad because it penalises large disagreements more and comes closer to human intuition.

Numerical calibration example

Imagine a golden set of 50 examples scored by a human and by the judge, both on a 1-5 scale. The confusion matrix:

	Hum 1	Hum 2	Hum 3	Hum 4	Hum 5	Judge total
Judge 1	3	1	0	0	0	4
Judge 2	1	4	2	0	0	7
Judge 3	0	1	6	2	0	9
Judge 4	0	0	1	12	3	16
Judge 5	0	0	0	2	12	14
Human total	4	6	9	16	15	50

Diagonal (exact agreements): 3+4+6+12+12 = 37 → p_o = 0.74.

p_e is computed as Σ_i (n_judge_i · n_hum_i) / n² = (4·4 + 7·6 + 9·9 + 16·16 + 14·15) / 50² = (16+42+81+256+210) / 2500 = 605/2500 ≈ 0.242.

$$\kappa = \frac{0.74 - 0.242}{1 - 0.242} = \frac{0.498}{0.758} \approx 0.66$$

Substantial. Acceptable. If we wanted quadratic weighted κ, near disagreements (Judge 4 vs Hum 5) weigh less than distant ones (Judge 2 vs Hum 4), and κ_quad typically comes out 0.05-0.10 above the linear one.

The four biases of the judge

Position bias

First documented by Wang et al. (2023). If you present two answers A and B to the judge in a pairwise setting, it prefers A more often than B, on the order of 55-65 % when A and B are objectively equivalent, with typical 2023-2024 judges. In 2026 frontier judges (GPT-5, Claude 4.5, Llama 4 Judge) have it fairly well mitigated, but it is still measurable.

How to measure it formally: run the dataset twice, once with (A, B) and once with (B, A). If the judge is consistent, the two passes should agree. The fraction of cases where the verdict flips is the position-bias rate. Accepted threshold in 2026: < 10 %.

Canonical mitigation: always run two passes with the order reversed and average. Frameworks such as Promptfoo and Inspect AI do it by default.

Verbosity bias

Documented by Saito et al. and Dubois et al. in 2024. For open-ended tasks (faithfulness, helpfulness), the judge tends to give a higher score to longer answers. The typical Pearson correlation between score and length on answers of equivalent human-rated quality can rise to 0.4-0.5 without mitigation.

How to measure it: Pearson correlation between the judge’s score and the length of the answer, computed over a subset where humans have confirmed equivalent quality. If the correlation is > 0.3 on that controlled subset, there is significant verbosity bias.

Mitigation: a rubric explicitly neutral on length (“the answer should be appropriately concise for the question; length is not a criterion”) and few-shot examples where a short answer beats a long one. AlpacaEval 2.0 builds a length correction directly into the metric.

Self-preference bias

Documented by Panickssery et al. (Anthropic, 2024). A GPT-4 judge prefers GPT-4 outputs. A Claude judge prefers Claude outputs. Not out of conspiracy, but because models share stylistic patterns with their close relatives (paragraph structure, use of bullet points, tone).

How to measure it: over a golden set, compare the scores of 3 different judges (e.g. GPT-4, Claude, Llama 4 Judge). If for the same candidate there is > 15 % systematic divergence attributable to the candidate’s identity, there is self-preference.

Mitigation: a panel of judges with heterogeneous vendors. If a single judge is going to be used, it must not come from the same vendor as the model being evaluated.

Narcissism

The extreme case of self-preference: the judge is exactly the same model as the candidate. This happens more than it seems: a team trains a Llama 3 8B with LoRA and evaluates it with Llama 3 8B as the judge because “it is what they have on-prem”. It is methodologically invalid. The delta between human score and judge score grows measurably.

Mitigation: a judge of a different architecture from the candidate. If your candidate is Llama 3, your judge should be Mistral, Qwen or a Prometheus 2 (which, although based on Mistral, was fine-tuned specifically for evaluation).

The judge as the pipeline’s hinge

The judge is not just “the eval piece”. It is the hinge that connects three stages of the pipeline:

Towards TUNE: the judge generates the (chosen, rejected) pairs for DPO without needing human labellers. That chain is what makes continuous fine-tuning work without a dedicated annotation team.

Towards the EVAL gate: the judge gives the score that is compared against the CI threshold. If the adapter does not clear 0.85 on faithfulness, no merge.

Towards continuous EVAL: over a sample of real traffic (1-5 %), the judge computes scores and persists them. That makes it possible to detect regressions that appear days after deployment and that CI did not see because its golden set did not cover them.

Towards RETRAIN: the cases where the judge gives a low score are automatic candidates for the next retraining dataset. The judge acts as the triage of the incident flow.

Implications for on-premise hardware

The numbers below are indicative for typical scenarios in May 2026.

On an RTX 4090 (24 GB)

Judge	Does it fit?	Approximate throughput	Notes
GPT-4o (API)	n/a	~50-100 judgements/min	External cost, not local
Prometheus 2 (8×7B INT4)	Yes, just about	~40-80 judgements/min	Q4_K_M GGUF, llama.cpp
Llama 3.1 8B fine-tuned judge	Yes, comfortably	~150-250 judgements/min	Reasonable on-prem default
Mistral Small Judge 22B	Not directly, requires offload	~10-20 judgements/min	Too much for 24 GB in BF16

Conclusion: on a single 4090, an open source 8B judge fine-tuned for evaluation (or a quantised Prometheus 2) is the sweet spot.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Judge	Configuration	Approximate throughput	Notes
Prometheus 2 BF16	TP=2	~400-700 judgements/min	Fits comfortably, low latency
Llama 3.3 70B Instruct	TP=4	~150-300 judgements/min	If used as a generalist judge
Panel of 3 judges in parallel	TP=1-2 each	~600-1200 judgements/min combined	Natural pattern in a cluster

On an NVLink cluster the natural approach is to run a panel of judges in parallel (each judge taking 1-2 GPUs) with a LiteLLM router in front. That removes the cognitive cost of “which judge do we use” because all three are used and the result is aggregated.

What we have not covered (upcoming articles)

Red teaming and safety eval: how robustness against adversarial prompts is evaluated. It is a different pattern from the ordinal judge.
Multistep agent eval: AgentBench, TauBench, evaluating trajectories instead of individual outputs.
Benchmark contamination: how to detect whether the evaluated model saw the golden set during pre-training, and why public benchmarks are half broken.
Cost-aware judging: when a cheap judge (Llama 8B) beats an expensive one (GPT-4o), and how to quantify the quality/cost trade-off with Pareto curves.

References

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (EMNLP 2023).
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., Seo, M. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (EMNLP 2024).
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Cohere, 2024).
Panickssery, A., Bowman, S., Feng, S. LLM Evaluators Recognize and Favor Their Own Generations (Anthropic, NeurIPS 2024).
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., Sui, Z. Large Language Models are Not Fair Evaluators (ACL 2024).
Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Stanford, 2024).
Cohen, J. A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement, 1960).
Landis, J. R., Koch, G. G. The Measurement of Observer Agreement for Categorical Data (Biometrics, 1977).

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

Mon, 25 May 2026 07:30:00 +0200

This is stage 3 of the LLMOps pipeline. If you arrive without context on the full journey, the six-stage LLMOps pipeline describes where Eval sits between Tune and Deploy, and the anatomy of a production LLM request shows a real eval blocking the promotion of an adapter.

TL;DR

Tracing is not evaluation. Tracing tells you what happened; eval tells you whether what happened is any good. The two layers live in tools that overlap visually (Langfuse does both), but the discipline is different: tracing is continuous capture over real traffic; eval is controlled execution against a stable dataset, with metrics that have to fail CI if they drop below a threshold. Without eval, the Tune → Deploy loop closes blind: adapter v8 goes to production because the engineer “saw it answering well” on five examples. With eval done properly, v8 only gets in if it beats v7 across a battery of 500 curated cases, evaluated by a mix of heuristics, embeddings, an LLM judge calibrated against humans, and a sample of real traffic with human annotation. This post takes apart the mechanism, the mathematics that keep you honest, the real tools in 2026 and the pitfalls that turn it into theatre.

The analogy: the academic examination board

A doctoral candidate defends a thesis. They do not defend it before a single distracted professor: they defend it before a mixed board, with a syllabus fixed in advance (not improvised on the day), and with an explicit pass mark that separates pass from fail. The board is not a single expert: it is a panel combining fast readers (the heuristics: does it have the requested format? does it include the citation?), semantic reviewers (the embeddings: does it resemble the expected answer?), an external assessor trained on the subject (the LLM judge: is it faithful to the context, is it relevant, does it sound coherent?), and human jury members on a sample of the most sensitive cases. If the candidate does not reach the mark, they are not promoted: they go back and prepare the defence again.

That analogy has three edges worth holding on to from the first minute:

The syllabus is fixed, but it is actively updated when a new topic appears in the real world that the candidate must know. If it is not updated, the exam measures something increasingly distant from what happens outside.
The board has to be trained: an LLM judge not calibrated against humans is an examiner making up the criteria.
The pass mark is published beforehand: it is not decided after seeing the result, because then it is not a pass mark, it is a justification.

These three ideas run through the rest of the post. Every tool and every piece of mathematics that follows is, at bottom, a way of operationalising them.

The mechanism itself: four layers of evaluators

An eval suite in 2026 is made up of four layers that coexist. None replaces the others; each one measures what the rest cannot measure well, and leaves unmeasured what the rest do measure well.

Layer 1 — Deterministic heuristics. Rules that return true or false with no ambiguity: the output matches a regex, contains a specific entity, does not exceed a length, follows a valid JSON schema, respects a requested format (markdown, tool_call function, mandatory citation). They are cheap, extremely fast, need neither judge nor embeddings, and catch the most frequent class of bug: the model answered in the wrong format. Their limit is obvious, in that they do not know whether the answer is correct, only whether it is well formed.

Layer 2 — Semantic metrics with embeddings. They compare the model’s output with an expected answer by computing cosine similarity between their embeddings, or by measuring whether a claim in the output is entailed by the retrieved context. They are cheap, deterministic given the embedding model, and very useful for detecting answers that drift in meaning but not in form. Their limit is equally clear: two answers can have high cosine similarity and say the opposite of each other (“the customer can cancel at any time” vs “the customer cannot cancel at any time” share 80% of their tokens).

Layer 3 — LLM-as-judge. A model, or an ensemble, evaluates the output of the model under test with a prompt designed to produce a score on a rubric. The canonical methods in 2026 are G-Eval (chain-of-thought prompting with a calibrated numeric score), Prometheus (open-source judge trained specifically for evals, reporting 0.897 correlation with humans in its v2.5 release from late 2025), and panel-of-judges setups that average votes from three heterogeneous models to reduce bias. This layer captures nuances the previous two cannot see: is it faithful to the context? is it useful? is it safe? is it complete? Its limit is cost and the need for calibration, treated as a whole section below.

Layer 4 — Humans. Trained annotators who evaluate a sample of the eval set, not all of it. They are the only layer with ultimate authority over the rubric: the LLM judge is calibrated against them, not the other way round. They are expensive (≈ 0.50–2.00 € per annotated sample when the domain is technical) and slow (a competent annotator produces 60–120 quality annotations per working day). The mistake teams make again and again is dropping this layer “because we have an LLM judge”; without humans there is no calibration, and without calibration the judge measures whatever it feels like.

Normal operation of an eval gate combines all four: heuristics remove malformed outputs before spending judge tokens, embeddings filter out the manifestly irrelevant, the judge scores the rest, and humans annotate a sample every N runs to keep the judge calibrated.

The golden dataset: a versioned syllabus

The golden dataset is the most undervalued artefact in the pipeline. It is the exam. If it is badly built, everything else, the best-calibrated judge in the world, the strictest gates, the fastest suite, measures noise. Covered at the top level in the data versioning post as one of the four artefacts to version differently; here we go into detail from the Eval perspective.

The three properties a golden dataset has to satisfy are:

Stratified representativeness. The dataset has to cover the system’s real input space in proportions that reflect production. If 30% of real traffic is in German, 30% of the golden set has to be in German; if 12% of questions are about subscription cancellation, that category cannot be 60% of the eval set just because it was easy to annotate. Stratification stays auditable: each example carries tags (lang=de, category=cancellation, tenant_type=enterprise, difficulty=hard) and the suite reports metrics per segment, not just the aggregate.

Strict holdout, no contamination with training. This rule is so obvious that almost every team believes it is following it, and almost every team breaks it without noticing. If the golden eval set gets mixed with the fine-tuning dataset, because somebody did a badly built random_split, because a purchased dataset is used by other vendors for training, because the LLM judge saw it during pretraining, then the metric stops measuring generalisation and starts measuring memorisation. The eval set hash is versioned separately (covered in the continuous fine-tuning post) and a routine leakage check is run: if an example in the golden set matches one in training token for token, it raises an alert.

Reasoned sample size, not aspirational. How many examples are needed? There is a mathematical minimum for telling two models apart with confidence. If model A is right 80% of the time and model B 85% on the same set, the 95% confidence interval for that difference of proportions (unpaired) is:

\[ \Delta p \pm 1{.}96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

To distinguish 80% from 85% with 95% confidence (an interval that does not cross zero), you need n ≈ 700 examples. To distinguish 90% from 91% under the same criterion, the calculation gives n ≈ 6,500. A “we have 50 examples in the golden set and we see adapter v8 scoring 90%” means nothing statistically: the confidence interval is ±8 points. The field’s practical rule in 2026 is a minimum of 300 examples to detect coarse differences, ideally 500–1,500 if you want to detect fine improvements, and to start by stratifying properly before obsessing over sample size.

On top of this comes active maintenance: the golden set is enriched with production incidents (covered in detail in the retrain post), so that every real complaint that ends in a bug becomes a curated example the next deploy candidate will have to pass. The golden set is not static: it is a living record of errors the system has already made and must not make again.

LLM-as-judge: how you calibrate an examiner

Layer 3 is the one most teams misuse. The typical mistake is: “we use GPT-4 as judge because it is the most capable”. The judge is not chosen for nominal capability; it is chosen for agreement with humans on the specific rubric you are measuring. A judge with 60% agreement is useless even if it is GPT-5; a Prometheus 7B fine-tuned for your domain with 88% agreement is worth more.

The standard metric for measuring agreement between two annotators (human vs judge, or two humans with each other) is Cohen’s kappa, which corrects for the agreement expected by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]

where $p_o$ is the observed proportion of agreement and $p_e$ is the proportion expected by chance under each annotator’s marginal distributions. The accepted interpretations in the literature are:

κ < 0.40: poor agreement. The judge says whatever it likes.
κ ∈ [0.40, 0.60]: moderate. Acceptable for coarse signals (is it toxic?), dreadful for nuance (is it faithful to the context?).
κ ∈ [0.60, 0.80]: substantial. Useful in production for most metrics.
κ > 0.80: almost perfect. The judge can be treated as a substitute for the human for that specific kind of judgement.

Numerically, consider a binary rubric (faithful / not faithful) over 200 examples annotated by a human and by a judge. If the human said “faithful” in 150 cases and the judge in 140, and they agree on 175 of the 200, then $p_o = 0{.}875$; the marginals are $p_h = 0{.}75$, $p_j = 0{.}70$, and $p_e = 0{.}75 \cdot 0{.}70 + 0{.}25 \cdot 0{.}30 = 0{.}600$. Kappa comes out at 0.6875: substantial but not excellent, usable, with an eye kept on the rubric.

Calibrating the judge means an explicit process:

Build a calibration set — 100–300 examples annotated by trained humans, with written guidelines. Inter-annotator kappa between the humans is also measured; if the humans themselves cannot agree, the rubric is badly written before you even start talking about a judge.
Iterate the judge prompt until judge agreement with the humans passes the accepted threshold (typically κ ≥ 0.7 for sensitive metrics).
Pin the judge version (claude-3-5-sonnet-20251022, gpt-4o-2024-11, prometheus-2-7b@sha256:…): any change invalidates the calibration.
Recalibrate periodically — every time the judge, the judge prompt, or the rubric changes. The frequency the field recommends in 2026 is quarterly at minimum, monthly if the rubric is new.
Persist everything in lineage — a score of “faithfulness 0.87” with no traceability of which judge, which prompt, which calibration set and which human validated it, is decorative.

The prompt versioning post covers how judge prompt versioning is materialised. The practical consequence is that the judge is versioned like any other model: your eval suite has judge_id = prometheus-2.5@v3 just as it has adapter_id = customer_support_v7.

The two cadences: CI gate and platform regression

Eval suites live in two places and run on two different cadences. Teams that conflate them turn one of the two into theatre.

CI gate (pre-merge, blocking). It runs on every pull request that modifies prompts, adapters, RAG configuration, or any artefact that could move the model’s output. It runs against the golden dataset versioned at the hash that is on main. The gate fails the merge if:

the critical metric drops more than X percentage points in absolute terms against the baseline (typically X = 2);
some safety metric (toxicity, PII leakage) crosses a hard threshold (typically tox > 0.02);
some strategic segment (a language, an enterprise-type tenant) drops more than Y points even if the aggregate improves.

This cadence has to be fast (ideally < 10 minutes over 500 examples) and cheap (LLM judge in batch mode, cached embeddings, heuristics run locally). The CI gate is not exhaustive: it is the low-latency line of defence.

Platform regression (post-deploy, continuous). It runs on a schedule (typically nightly or weekly) over sampled production traffic, not over the static golden set. It detects drift: the model has not changed, the golden set has not changed, but the users have, and quality over real traffic drops. This cadence is more expensive (judge over thousands of samples, human annotation over hundreds), tolerates latencies of hours, and its main consumer is not CI but the observability dashboard and the humans on the product team who decide whether to open a retrain cycle.

Both cadences persist results in the same store (Langfuse, MLflow, or equivalent) and connect them by model_id, prompt_id, dataset_hash and judge_id. Without that glue of identifiers, the metric that passes CI cannot be correlated with the one that fails in production three weeks later.

The minimum mathematics that matter

Beyond kappa and the interval for the difference of proportions, both already covered above, there are three more mathematical pieces that any team running evals seriously ends up using.

Confidence interval for a continuous metric. If your metric is a continuous score (faithfulness ∈ [0, 1]) and you measure the sample mean $\bar{x}$ over n examples with standard deviation s, the 95% confidence interval for the population mean is:

\[ \bar{x} \pm 1{.}96 \cdot \frac{s}{\sqrt{n}} \]

For n = 300 and s ≈ 0.2 (typical of a 0-1 score with non-degenerate variance), the margin is ±0.023. This means that differences below 2 hundredths cannot be distinguished from noise at that sample size. If your team is chasing “+0.5 pp” improvements over 100 examples, it is optimising noise.

Judge cost as a function of sample size and rubric. The cost of one eval pass with LLM-as-judge over n examples, with m metrics evaluated in a single call per example, and per-token price $c_{in}, c_{out}$ on the judge model, is:

\[ C \approx n \cdot (t_{in} \cdot c_{in} + t_{out} \cdot c_{out}) \]

where $t_{in}$ is the number of input tokens (includes context, output of the model under test, full rubric) and $t_{out}$ the output tokens (includes the judge’s CoT + score). For n = 500, $t_{in}$ ≈ 4,000, $t_{out}$ ≈ 300, a GPT-4o judge at May 2026 prices, one pass costs on the order of 8–15 USD per suite. If the pass fires on every PR and there are 30 PRs/day, that is 240–450 USD/day on CI gates alone. Multiplied by continuous regression, teams that do not control this spend four figures a month on judge tokens without realising. The canonical mitigation is a mix of layers: heuristics and embeddings filter first, the judge is only invoked on what the cheap layers cannot resolve, and for platform regression a self-hosted open-source judge is used (Prometheus 7B on your own GPU plane) instead of a commercial model.

Distinction between aggregate metric and per-segment metric. The classic eval fallacy is the hidden mean. If your suite reports faithfulness = 0.87 and the team reads it as “up 2 points against the previous adapter”, this may be what is happening: the new adapter gains 4 points in English (where 70% of the eval set is) and loses 6 points in German (where 30% is). The aggregate mean improves, the German experience gets worse. Any serious suite reports a breakdown by strategic segment (language, tenant type, question category, context length). The CI gate can also have per-segment thresholds, not just aggregate ones.

The 2026 stack: dominant tools

Tool	Main layer	Licence	Maintainer	When to pick it
DeepEval	CI gate	Apache 2.0	Confident AI	“Evals as pytest” — assertions in Python code, trivial integration with GitHub Actions. A reasonable default.
Promptfoo	CI gate	MIT	Promptfoo Inc.	Declarative YAML, matrix of prompts × providers × assertions, diff vs baseline. DevOps-friendly.
RAGAS	RAG-specific metrics	Apache 2.0	Exploding Gradients	Faithfulness, context relevancy, answer relevancy. The canonical piece if your system is RAG.
Inspect AI	Safety/capability evals	MIT	UK AI Safety Institute	Suite focused on safety and capability. Useful for regulatory gates under the EU AI Act.
Langfuse Evals	Platform regression	MIT (OSS) / EE	Langfuse GmbH	Integrated with tracing — datasets, runs and scores in the same UI as production traces.
MLflow GenAI Evals	Registry + evals	Apache 2.0	Databricks/LF AI	Good when you already have MLflow for classic models; “GenAI dashboard” since 3.10.
Phoenix (Arize)	Eval + visual drift	Elastic License 2.0	Arize AI	Focused on visual debugging of embeddings and drift; a complement to Langfuse, not a replacement.
Prometheus 2.5	Self-hosted OSS judge	Apache 2.0	KAIST + LG AI	Fine-tuned judge, high correlation with GPT-4 at zero cost per token when self-hosted.
G-Eval / JudgeLM	Prompting methods	— (techniques)	academic	Prompting frameworks for LLM-as-judge — they apply on top of any judge model.

The OSS catalogue by stage goes into detail entry by entry; the parallel OSS vs hyperscalers catalogue compares against Bedrock Evaluations, Vertex AI Eval Service and Azure AI Evaluation.

The canonical pattern in 2026 is hybrid: DeepEval or Promptfoo for CI gates + Langfuse Evals for platform regression + Prometheus 2.5 as a self-hosted judge + human annotation on Argilla or Label Studio for the calibration set. Swapping any of these pillars for equivalents (W&B Weave instead of Langfuse, Inspect instead of DeepEval) is style, not functionality; what matters is that the four functions are present and connected.

Applied to typical on-premise hardware

For an on-premise deployment that wants to avoid sending sensitive outputs to a commercial LLM judge (for data sovereignty, ENS, NIS2 or equivalents), the judge is hosted on your own GPU plane. Reference figures for May 2026, based on Prometheus 2.5 (Llama-3.1-8B fine-tuned as a judge) served on vLLM:

RTX 4090 (24 GB, Ada Lovelace): viable for small suites (< 200 examples) and for running the judge offline during development. Latency per evaluation ≈ 1.5–3 s with BF16; aggregate throughput on the order of 25–40 evaluations/min with batching. Useful for a developer’s local CI gates, not for platform regression.
Generic 4×H100 SXM configuration (320 GB total, NVLink): runs the judge in parallel at TP=2 across two GPUs, leaving two free to serve the model under test. Aggregate throughput on the order of 200–350 evaluations/min, a full 500-example suite in 2–3 min. This allows PR gates with no perceptible waiting and nightly platform regression over thousands of samples with no per-token cost.

The comparative cost arithmetic is stubborn: self-hosting Prometheus 2.5 pays off an H100 in roughly 6 months if the team fires ≥ 30 PRs/day with eval gates. Below that volume, the commercial judge still wins unless sovereignty is a requirement, and under ENS / NIS2 it is.

The seven pitfalls that kill this stage

Pitfall 1 — Aged golden dataset. It is not enriched with production incidents. After months it measures a world that no longer exists. The metric climbs comfortably while real users complain more.

Pitfall 2 — Contaminated or uncalibrated judge. The LLM judge evaluates with criteria it invents itself. Without a human calibration set as reference, there is no way to know whether its 0.89 is generous, severe or random.

Pitfall 3 — Insufficient sample size. A 50-example suite, 1-point differences the team treats as significant. The confidence interval is ±10 points. They are measuring noise and making real decisions on it.

Pitfall 4 — Runaway cost. A GPT-4 batch-mode judge fired on every PR over 1,000 examples, with no prior filtering by cheap layers. The eval bill overtakes production serving. It happens more often than people admit.

Pitfall 5 — Aggregate metric hiding segments. Global mean improves 2 points, German drops 6, enterprise tenants drop 3. With no explicit per-segment breakdown, the gate approves what it should not.

Pitfall 6 — Judge with a floating version. Judge model updated without recalibrating the rubric. Thresholds lose their statistical meaning. Last month’s regressions are not comparable with those from two months ago.

Pitfall 7 — Eval gate that is not enforced. The gate exists in the documentation but not in the real workflow: the suite takes 30 minutes, developers skip it with --no-verify, managers ask for one-off exceptions that become the norm. An eval gate without enforcement is ornament.

All seven are operational, not technical. The Eval layer does not break because the mathematics are wrong: it breaks because the discipline slackens. It is the same thing that happens with unit tests in any project that grows, except that here, without the discipline, the system improves its metrics while getting worse, and that makes the degradation invisible until it is already ungovernable.

What we have not covered (upcoming posts)

Guardrails and online safety: the inline eval layer that filters outputs in real time, not in CI. Conceptually a cousin of Eval, but with very different latency constraints. Covered in detail in the guardrails post: four lines of defence (input, retrieval, tool, output), OWASP LLM Top 10 (2025), OSS catalogue and deployment patterns.
Judge ensembles and agreement between multiple judges: how to reduce bias by combining three heterogeneous models as a panel, which aggregation function works (simple majority, trimmed mean, calibrated meta-judge).
Meta-eval: how you evaluate the suite itself. If the eval improvement from adapter v7 to v8 does not translate into a real improvement for the user, the suite is wrong, and that is also measured, by correlating eval metrics against product metrics.
Metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, alternative language) as an extra gate. Beyond nominal agreement, it measures consistency.
Adversarial evals with red teaming: introducing prompt injection and jailbreak attacks as part of the gate.
Privacy in the judge: how to stop sensitive outputs from the model under test travelling to an external judge when regulation forbids it — homomorphic judge, judge in a TEE, or simply a self-hosted judge.

References

G-Eval: Liu et al., “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” — the reference paper for the chain-of-thought LLM judge method.
Prometheus: Kim et al., “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models” — OSS judge with a reported 0.897 correlation vs humans.
RAGAS: Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation” — the paper that standardises faithfulness, context relevancy and answer relevancy.
Cohen’s kappa: Cohen, J. (1960). “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement — the classic metric for inter-annotator agreement, still the operational reference.
DeepEval docs: https://docs.confident-ai.com/
Promptfoo docs: https://promptfoo.dev/docs/
Langfuse Evals: https://langfuse.com/docs/scores
Inspect AI: https://inspect.ai-safety-institute.org.uk/
EU AI Act, relevant articles on mandatory evaluation of high-risk systems — pending publication of the CEN/CENELEC technical standards on conformity assessment for GenAI.

Evals on lo0 — Tech Blog

LLM-as-judge: the exam marker who grades other models without turning into an oracle

TL;DR

You are here: EVAL

The analogy: the civil service exam marker

Why LLM-as-judge exists

The three canonical patterns in 2026

1 · G-Eval

2 · Prometheus 2

3 · Panel of Judges

How to measure whether the judge lies: Cohen’s kappa

Weighted kappa for ordinal scales

Numerical calibration example

The four biases of the judge

Position bias

Verbosity bias

Self-preference bias

Narcissism

The judge as the pipeline’s hinge

Implications for on-premise hardware

On an RTX 4090 (24 GB)

On a generic 4×H100 SXM cluster (320 GB, NVLink)

What we have not covered (upcoming articles)

See also

References

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

TL;DR

The analogy: the academic examination board

The mechanism itself: four layers of evaluators

The golden dataset: a versioned syllabus

LLM-as-judge: how you calibrate an examiner

The two cadences: CI gate and platform regression

The minimum mathematics that matter

The 2026 stack: dominant tools

Applied to typical on-premise hardware

The seven pitfalls that kill this stage

What we have not covered (upcoming posts)

See also

References