LLM-as-judge: the exam marker who grades other models without turning into an oracle

Contents

This post goes deeper into the judges section of Evals for LLMs. There it was one piece of the mixed panel; here we get into why it works, where it breaks and how its calibration is measured before accepting it in CI.

TL;DR

An LLM judge is not “a GPT-4 you ask whether the answer is any good”. It is a trained marker: it has a rubric written in advance, it is asked for explicit reasoning before the verdict, its score is computed as an expectation weighted by token probabilities (not as the first token it spits out), and before being accepted in production it is calibrated against a sample of ~50 human-annotated examples until it reaches κ ≥ 0.5 (Cohen’s kappa). The state of the art in May 2026 is three patterns, G-Eval, Prometheus 2 and panel of judges, each answering a different trade-off between cost, quality and reproducibility. All of them share four documented biases: position, verbosity, self-preference and narcissism. This post explains how a real judge is built, how to measure whether it lies, and when each pattern is appropriate.

You are here: EVAL

The analogy: the civil service exam marker

A public competitive exam has thousands of papers. There is no way the senior board can mark them all. The Spanish system’s answer has been the same for decades: trained markers. People who are not professors, but who are given:

A marking template written in advance: what is valued, how many points each section is worth, what gets deducted.
Prior training with a sample of papers already marked by the senior board, until their marks agree reasonably well.
Continuous auditing: a fraction of their marking is re-marked by the senior board to verify that the marker is not drifting.

A well-built LLM judge is exactly that. It is not an oracle, it is a trained marker:

The rubric is the explicit criteria in the prompt (what faithfulness is, what relevancy is, and so on).
The training is the calibration against ~50 human-annotated examples.
The continuous auditing is the weekly sampling where a human re-evaluates a fraction of the traffic the judge has judged.

And, as with the human marker, the judge has systematic biases that the public exam system has learned to watch for: markers prefer certain handwriting styles, certain lengths, certain structures. The same happens to the judge. The rest of this post takes apart exactly which ones and how they are measured.

Why LLM-as-judge exists

The direct reason: money and time. A professional human annotation costs on the order of 0.50 € to 5 € per example (depending on complexity and domain), and takes 30 seconds to several minutes. A GPT-4 judgement costs ~0.01-0.05 € and takes ~2 seconds. For a golden dataset of 500 examples evaluated continuously over 10 candidates a day, the difference is between 2,500 € a day and 50 € a day. And the wall-clock difference is between days and minutes.

The less direct but more relevant reason: methodological scalability. A golden dataset of 500 examples is relatively easy to annotate once. What happens afterwards is the hard part:

Every time a candidate adapter appears, all 500 have to be re-evaluated.
Every time the dataset is updated (because a new incident came in) it has to be re-evaluated.
Every time the system prompt changes it has to be re-evaluated.

Without an automatable, cheap judge, the eval battery stops running and the system goes blind between releases. That is what really justifies the pattern.

The three canonical patterns in 2026

1 · G-Eval

Published by Liu et al. (2023). The base idea is so simple it fits in one sentence: give the judge a detailed rubric, ask it to reason before scoring, and read the result as a weighted expectation instead of as the first token. The three levers:

Rubric: a prompt with an explicit criterion (e.g. “Faithfulness: the degree to which the answer relies only on the context, without inventing data. Score 1 = invents everything, 5 = everything supported by the context”), ideally with one or two examples per extreme value.

Chain-of-thought + form-filling: the model is first asked to “reason briefly about each criterion” and then to “fill in this JSON form”. That forces it not to spit out an arbitrary number.

Probability-weighted scoring: instead of reading the first token after the score: field, you look at the model’s probability distribution over the tokens 1, 2, 3, 4, 5 and compute:

$$\hat{s} = \sum_{i=1}^{5} i \cdot \frac{p(\text{token}=i)}{\sum_{j=1}^{5} p(\text{token}=j)}$$

This turns a discrete score into a continuous one. The justification: if the judge was “torn” between 4 and 5 (probabilities 0.4 and 0.5 on 4 and 5), the real score is 4.55, not 5. This sharply reduces variance across runs of the same prompt, and captures information that greedy decoding throws away.

Limitations of G-Eval: it needs access to the model’s logprobs. Closed-source models have been restricting that access (Claude does not expose it, GPT-4 does but only top-5). In 2026 G-Eval with strict probability weighting is only practical against open source models you serve yourself (vLLM exposes them) or against GPT-4 with logprobs=true.

2 · Prometheus 2

Published by Kim et al. (KAIST, 2024). The insight is complementary to G-Eval: what if, instead of asking a generalist judge to evaluate, we fine-tune a specific judge?

Prometheus 2 is a Mistral 8×7B (MoE, ~47 GB in BF16, ~24 GB in INT4) fine-tuned on 100k+ evaluation examples with varied rubrics. The metric published in the paper: 0.897 Pearson correlation with GPT-4-as-judge on the Vicuna Bench and similar. That matters because it means GPT-4 can be replaced as a judge at the cost of local inference without losing almost any quality.

Why it matters in on-premise production:

No data leaves the perimeter. For strict ENS/NIS2 customers this is not a preference, it is a requirement. A judge that travels over an external API is not an option there.
Zero marginal cost. When the judge runs on-prem, evaluating 50,000 cases a day adds no external bill.
Controlled latency. Continuous eval over real traffic can run in parallel without saturating an external provider’s rate limits.

The price: you have to maintain one more inference service (Prometheus 2 running in its own vLLM), and the judge does not “update” unless it is re-fine-tuned.

3 · Panel of Judges

Verga et al. (Cohere, 2024) formalised it: instead of a single judge, use 3-5 heterogeneous judges, different models, different prompts, different temperatures, and aggregate their judgements.

Common aggregation mechanisms:

Median for continuous scores. Robust to outliers.
Majority vote for pairwise judgements (chosen vs rejected). If 3 of 5 prefer A, the winner is A.
Calibration-weighted mean: weight each judge by its κ against humans in calibration. The more reliable judges vote more.

What a panel gives that a single judge does not:

Reduction of self-preference (a judge’s bias towards outputs stylistically similar to its own): if the judges come from different vendors, the sum cancels out.
A measure of case difficulty: if all 5 judges agree, the case is easy; if they are split, the case is ambiguous and should be escalated to a human. That turns the panel into an automatic triaging system for human annotation.
Lower variance: each judge’s noise is averaged out.

The cost: 3-5× the bill of a single judge. That is why the panel is usually reserved for critical eval gates (does this adapter get promoted or not?), not for continuous eval over traffic.

How to measure whether the judge lies: Cohen’s kappa

Accepting the judge in production without measuring its agreement with humans is the same as accepting an uncalibrated thermometer. The standard metric for inter-rater agreement with discrete or ordinal scales is Cohen’s kappa:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Where p_o is the observed agreement proportion (what percentage of examples judge and human agree on) and p_e is the agreement proportion expected by chance (what you would expect if both scored at random respecting each one’s marginals).

The intuition: if p_o = 0.9 but p_e = 0.85 (because both nearly always score “4 or 5”), agreement is 90 % in raw terms but κ = 0.33: most of the agreement comes from both scoring high, not from them understanding each other. κ corrects for that baseline.

The usual interpretive scale (Landis and Koch 1977, still the reference):

κ	Interpretation	Production threshold
< 0.20	Poor	Useless
0.21–0.40	Fair	Weak signal only
0.41–0.60	Moderate	Minimum acceptable in 2026
0.61–0.80	Substantial	State of the art for open source judges
0.81–1.00	Almost perfect	Human judges rarely reach it between themselves

An important point the field learned the hard way: humans between themselves rarely exceed κ = 0.70 on LLM tasks (faithfulness, relevancy). That is the realistic ceiling for an LLM judge. Chasing κ = 0.9 against humans is chasing a ghost: not even two human annotators get there.

Weighted kappa for ordinal scales

For 1-5 scores, the disagreement “judge says 4, human says 5” is not the same as “judge says 1, human says 5”. Standard kappa treats both as an identical failure. Linear or quadratic weighted kappa assigns weight to the magnitude of the disagreement:

$$\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} \, o_{ij}}{\sum_{i,j} w_{ij} \, e_{ij}}, \quad w_{ij}^{(\text{lin})} = \frac{|i-j|}{k-1}, \quad w_{ij}^{(\text{quad})} = \frac{(i-j)^2}{(k-1)^2}.$$

In G-Eval with 1-5 scores, the usual practice is to publish κ_quad because it penalises large disagreements more and comes closer to human intuition.

Numerical calibration example

Imagine a golden set of 50 examples scored by a human and by the judge, both on a 1-5 scale. The confusion matrix:

	Hum 1	Hum 2	Hum 3	Hum 4	Hum 5	Judge total
Judge 1	3	1	0	0	0	4
Judge 2	1	4	2	0	0	7
Judge 3	0	1	6	2	0	9
Judge 4	0	0	1	12	3	16
Judge 5	0	0	0	2	12	14
Human total	4	6	9	16	15	50

Diagonal (exact agreements): 3+4+6+12+12 = 37 → p_o = 0.74.

p_e is computed as Σ_i (n_judge_i · n_hum_i) / n² = (4·4 + 7·6 + 9·9 + 16·16 + 14·15) / 50² = (16+42+81+256+210) / 2500 = 605/2500 ≈ 0.242.

$$\kappa = \frac{0.74 - 0.242}{1 - 0.242} = \frac{0.498}{0.758} \approx 0.66$$

Substantial. Acceptable. If we wanted quadratic weighted κ, near disagreements (Judge 4 vs Hum 5) weigh less than distant ones (Judge 2 vs Hum 4), and κ_quad typically comes out 0.05-0.10 above the linear one.

The four biases of the judge

Position bias

First documented by Wang et al. (2023). If you present two answers A and B to the judge in a pairwise setting, it prefers A more often than B, on the order of 55-65 % when A and B are objectively equivalent, with typical 2023-2024 judges. In 2026 frontier judges (GPT-5, Claude 4.5, Llama 4 Judge) have it fairly well mitigated, but it is still measurable.

How to measure it formally: run the dataset twice, once with (A, B) and once with (B, A). If the judge is consistent, the two passes should agree. The fraction of cases where the verdict flips is the position-bias rate. Accepted threshold in 2026: < 10 %.

Canonical mitigation: always run two passes with the order reversed and average. Frameworks such as Promptfoo and Inspect AI do it by default.

Verbosity bias

Documented by Saito et al. and Dubois et al. in 2024. For open-ended tasks (faithfulness, helpfulness), the judge tends to give a higher score to longer answers. The typical Pearson correlation between score and length on answers of equivalent human-rated quality can rise to 0.4-0.5 without mitigation.

How to measure it: Pearson correlation between the judge’s score and the length of the answer, computed over a subset where humans have confirmed equivalent quality. If the correlation is > 0.3 on that controlled subset, there is significant verbosity bias.

Mitigation: a rubric explicitly neutral on length (“the answer should be appropriately concise for the question; length is not a criterion”) and few-shot examples where a short answer beats a long one. AlpacaEval 2.0 builds a length correction directly into the metric.

Self-preference bias

Documented by Panickssery et al. (Anthropic, 2024). A GPT-4 judge prefers GPT-4 outputs. A Claude judge prefers Claude outputs. Not out of conspiracy, but because models share stylistic patterns with their close relatives (paragraph structure, use of bullet points, tone).

How to measure it: over a golden set, compare the scores of 3 different judges (e.g. GPT-4, Claude, Llama 4 Judge). If for the same candidate there is > 15 % systematic divergence attributable to the candidate’s identity, there is self-preference.

Mitigation: a panel of judges with heterogeneous vendors. If a single judge is going to be used, it must not come from the same vendor as the model being evaluated.

Narcissism

The extreme case of self-preference: the judge is exactly the same model as the candidate. This happens more than it seems: a team trains a Llama 3 8B with LoRA and evaluates it with Llama 3 8B as the judge because “it is what they have on-prem”. It is methodologically invalid. The delta between human score and judge score grows measurably.

Mitigation: a judge of a different architecture from the candidate. If your candidate is Llama 3, your judge should be Mistral, Qwen or a Prometheus 2 (which, although based on Mistral, was fine-tuned specifically for evaluation).

The judge as the pipeline’s hinge

The judge is not just “the eval piece”. It is the hinge that connects three stages of the pipeline:

Towards TUNE: the judge generates the (chosen, rejected) pairs for DPO without needing human labellers. That chain is what makes continuous fine-tuning work without a dedicated annotation team.

Towards the EVAL gate: the judge gives the score that is compared against the CI threshold. If the adapter does not clear 0.85 on faithfulness, no merge.

Towards continuous EVAL: over a sample of real traffic (1-5 %), the judge computes scores and persists them. That makes it possible to detect regressions that appear days after deployment and that CI did not see because its golden set did not cover them.

Towards RETRAIN: the cases where the judge gives a low score are automatic candidates for the next retraining dataset. The judge acts as the triage of the incident flow.

Implications for on-premise hardware

The numbers below are indicative for typical scenarios in May 2026.

On an RTX 4090 (24 GB)

Judge	Does it fit?	Approximate throughput	Notes
GPT-4o (API)	n/a	~50-100 judgements/min	External cost, not local
Prometheus 2 (8×7B INT4)	Yes, just about	~40-80 judgements/min	Q4_K_M GGUF, llama.cpp
Llama 3.1 8B fine-tuned judge	Yes, comfortably	~150-250 judgements/min	Reasonable on-prem default
Mistral Small Judge 22B	Not directly, requires offload	~10-20 judgements/min	Too much for 24 GB in BF16

Conclusion: on a single 4090, an open source 8B judge fine-tuned for evaluation (or a quantised Prometheus 2) is the sweet spot.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Judge	Configuration	Approximate throughput	Notes
Prometheus 2 BF16	TP=2	~400-700 judgements/min	Fits comfortably, low latency
Llama 3.3 70B Instruct	TP=4	~150-300 judgements/min	If used as a generalist judge
Panel of 3 judges in parallel	TP=1-2 each	~600-1200 judgements/min combined	Natural pattern in a cluster

On an NVLink cluster the natural approach is to run a panel of judges in parallel (each judge taking 1-2 GPUs) with a LiteLLM router in front. That removes the cognitive cost of “which judge do we use” because all three are used and the result is aggregated.

What we have not covered (upcoming articles)

Red teaming and safety eval: how robustness against adversarial prompts is evaluated. It is a different pattern from the ordinal judge.
Multistep agent eval: AgentBench, TauBench, evaluating trajectories instead of individual outputs.
Benchmark contamination: how to detect whether the evaluated model saw the golden set during pre-training, and why public benchmarks are half broken.
Cost-aware judging: when a cheap judge (Llama 8B) beats an expensive one (GPT-4o), and how to quantify the quality/cost trade-off with Pareto curves.

References

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (EMNLP 2023).
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., Seo, M. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (EMNLP 2024).
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Cohere, 2024).
Panickssery, A., Bowman, S., Feng, S. LLM Evaluators Recognize and Favor Their Own Generations (Anthropic, NeurIPS 2024).
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., Sui, Z. Large Language Models are Not Fair Evaluators (ACL 2024).
Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Stanford, 2024).
Cohen, J. A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement, 1960).
Landis, J. R., Koch, G. G. The Measurement of Observer Agreement for Categorical Data (Biometrics, 1977).