Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

Wed, 27 May 2026 08:00:00 +0200

This post is the natural continuation of continuous fine-tuning in production, which covers the operational cycle (Postgres + SQL queries + hot-swap) that feeds these methods with real data. Here we go inside each one: what each loss optimises, what hypothesis it makes, and why they choose one or the other.

TL;DR

Classic RLHF, the kind in the InstructGPT papers, is practically extinct in production. The reason is not ideological: it is that in 2023 Rafailov and others showed that the reward model does not need to exist as a separate object. With an elegant change of variable, the optimal policy can be trained directly from preference pairs, without going through a reward and without RL. That is DPO. From there the family has branched into four methods that coexist in 2026: DPO when you have (chosen, rejected) pairs, KTO when you only have a binary 👍/👎 signal, ORPO when you want SFT and preferences in the same pass to save memory, and SimPO when you also want to get rid of the reference model and normalise by length. This post explains exactly what each loss does, proves it with an end-to-end numerical example, gives the real decision table between the four and takes apart the three biases that kill the method in production when nobody watches them.

You are here: TUNE

The six-stage LLMOps pipeline places Tune between Data and Eval. Inside Tune three modalities coexist: SFT (supervised fine-tuning), preference optimization (what this post covers) and agent training / RFT. What follows is the inside of the second modality.

The analogy: the sommelier who trains the palate without a theory book

Imagine you want to train a sommelier. You have two routes. The first is teaching them oenological theory: grape varieties, terroirs, vinification methods, barrel types. You give them cards with canonical wine descriptions and ask them to memorise them. That is SFT: (prompt, ideal answer) pairs. It works as general education, but the sommelier who comes out of it cannot tell two excellent Riojas apart.

The second route is the blind comparator. You put two identical opaque glasses in front of them. You say “this one is better than this one”. You do not explain why. You repeat the exercise a thousand times, with a thousand different pairs. After a while, the sommelier does not need you to say it: they have a trained palate. They have not learned oenological theory, they have learned to discriminate.

DPO, KTO, ORPO and SimPO are four variants of the second route. All four train the model to discriminate, not to memorise. The differences between them are:

DPO: two glasses at each tasting, the sommelier always knows which one is the “good” one and which is the “bad” one.
KTO: a single glass each time, they are only told “you would like this” or “you would not like this”. No pairs.
ORPO: the tasting also includes a small embedded theory class (SFT in parallel) so the catalogue is not forgotten.
SimPO: like DPO, but the sommelier does not compare against “what your master would have said” (reference model); they compare the two glasses directly, normalising by the amount of liquid in each.

The analogy is not decorative: the rest of the post is that same idea expressed mathematically.

Why DPO exists: Rafailov’s trick

To understand DPO it helps to walk through in thirty seconds what it replaces. Classic RLHF, the InstructGPT kind, has three phases:

Phase 1 – SFT. You train the model on (prompt, ideal answer). Out comes π_ref: the reference policy. It is the “educated” model.

Phase 2 – Reward model. On the same model (a different head), you train a regressor: given (prompt, chosen, rejected), it learns to give a higher score to the chosen. Out comes r_φ(x, y).

Phase 3 – PPO. You take π_ref as the starting point and train another copy π_θ so that it maximises r_φ(x, π_θ(x)) with a KL penalty to keep it from drifting too far from π_ref. That requires generating rollouts (on-policy decode at every step), keeping the three models in memory (π_θ, π_ref, r_φ), and a classic RL setup: unstable, sensitive to hyperparameters and famous for its reproducibility problems.

Rafailov’s observation was: phase 3 has a closed-form solution. If you write out the exact optimisation problem PPO solves

$$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D, \, y \sim \pi_\theta(\cdot|x)} \big[ r_\phi(x,y) \big] - \beta \, \mathrm{KL}\big(\pi_\theta(\cdot|x) \,\|\, \pi_\mathrm{ref}(\cdot|x)\big)$$

it turns out the optimal policy has the form

$$\pi^{*}(y|x) = \frac{1}{Z(x)} \, \pi_\mathrm{ref}(y|x) \, \exp\!\left( \tfrac{1}{\beta} r_\phi(x,y) \right)$$

and from there you can solve for the implicit reward:

$$r_\phi(x,y) = \beta \log \frac{\pi^{*}(y|x)}{\pi_\mathrm{ref}(y|x)} + \beta \log Z(x).$$

The second term is a function of x only and cancels out when you compare two answers to the same prompt. The reward does not need to be learned: it is implicit in the log-prob ratio between the trained model and the reference one. If you plug that into the Bradley-Terry model for preferences (the formula that says “the probability that yw is preferred to yl is σ(r(x,yw) - r(x,yl))”), out comes the DPO loss:

$$\mathcal{L}_\mathrm{DPO}(\pi_\theta;\pi_\mathrm{ref}) = -\,\mathbb{E}_{(x, y_w, y_l) \sim D} \log \sigma\!\Big( \beta \big[ \log \tfrac{\pi_\theta(y_w|x)}{\pi_\mathrm{ref}(y_w|x)} - \log \tfrac{\pi_\theta(y_l|x)}{\pi_\mathrm{ref}(y_l|x)} \big] \Big).$$

That is the whole of DPO. There is no reward model, no RL, no rollouts. Only log-probs over static data.

DPO with real numbers

The formula is less intimidating when you evaluate it with an example. Imagine a pair from the dataset:

x = “Explain what a KVM switch is”.
y_w = correct answer (chosen).
y_l = confusing answer (rejected).

After a forward pass we have four numbers (sums of per-token log-probs, negative sign because each log p_token ≤ 0):

Quantity	Value
`log π_θ(y_w \| x)`	−45.2
`log π_θ(y_l \| x)`	−52.1
`log π_ref(y_w \| x)`	−47.3
`log π_ref(y_l \| x)`	−50.8

The “ratios” are the improvement of the trained model over the reference one for each answer:

For y_w: −45.2 − (−47.3) = +2.1 → the trained model gives it more probability than the reference one. Good.
For y_l: −52.1 − (−50.8) = −1.3 → the trained model gives it less probability. Also good.

With β = 0.1 (a typical value) the “margin” inside the logarithm is:

$$m = \beta \cdot (2.1 - (-1.3)) = 0.1 \cdot 3.4 = 0.34$$

And the loss:

$$\mathcal{L}_\mathrm{DPO} = -\log \sigma(0.34) = -\log(0.584) \approx 0.538.$$

The intuition is visible straight away: the more the chosen log-prob rises and the more the rejected one falls (relative to π_ref), the more positive m becomes, the higher the sigmoid and the lower the loss. If the margin is negative (the model gets it wrong), σ(m) < 0.5 and the loss blows up. The gradient pushes the model to increase π_θ(y_w|x) and lower π_θ(y_l|x).

The role of β: it is the temperature of the alignment. If β is small, the model is allowed to drift a long way from π_ref; if it is large, the KL weighs heavily and the model barely moves. Typical value in 2026: 0.05–0.3, with 0.1 as a starting point.

KTO: when you only have 👍/👎

DPO needs pairs. In practice, that is almost never what the product gives you: what the product gives you is binary feedback (a thumbs up or a thumbs down, a conversion or an abandonment). KTO — Kahneman-Tversky Optimization, Ethayarajh et al. 2024 — solves exactly that case.

The intuition comes from Kahneman and Tversky’s prospect theory: humans are more sensitive to losses than to gains of the same magnitude (loss aversion: losing 100 € hurts more than gaining 100 €). KTO carries that over into the loss:

$$\mathcal{L}_\mathrm{KTO}(x, y) = \begin{cases} \lambda_d \big[ 1 - \sigma\!\big(\beta \cdot ( h_\theta(x,y) - z_0 ) \big) \big] & \text{if } y \text{ is desirable},\\ \lambda_u \big[ 1 - \sigma\!\big(\beta \cdot ( z_0 - h_\theta(x,y) ) \big) \big] & \text{if } y \text{ is undesirable}, \end{cases}$$

where h_θ(x,y) = log(π_θ(y|x) / π_ref(y|x)) is exactly the same ratio that appeared in DPO, z_0 is an estimate of the batch KL divergence (it acts as a “neutral point”) and λ_d, λ_u are the desirable / undesirable weights.

The critical point: KTO does not need pairs. Each example is (prompt, answer, binary label). This fits real product telemetry. The usual practical rule is λ_u > λ_d (e.g. 1.0 vs 0.33) when the database has more 👍 than 👎, so that the negative signal is not diluted.

KTO works particularly well in two scenarios:

Products with explicit feedback UX (chatbots with 👍/👎): each thumbs is a direct KTO example, no need to synthesise pairs.
Imbalanced datasets (far more 👍 than 👎, or the other way round): the weights λ_d, λ_u handle it explicitly.

ORPO: SFT and preferences in a single pass

DPO assumes you have already done SFT. You train two phases: first SFT to get π_ref, then DPO on top of π_ref. Two passes over the data, two optimisations, two models in memory.

ORPO — Odds Ratio Preference Optimization, Hong et al. 2024 — merges both phases. The loss combines two terms:

$$\mathcal{L}_\mathrm{ORPO} = \mathcal{L}_\mathrm{SFT}(y_w) + \lambda \cdot \mathcal{L}_\mathrm{OR}(y_w, y_l)$$

The first term is classic SFT on the chosen answer (negative cross-entropy). The second is the odds ratio between chosen and rejected:

$$\mathcal{L}_\mathrm{OR} = -\log \sigma\!\Big( \log \tfrac{\mathrm{odds}_\theta(y_w|x)}{\mathrm{odds}_\theta(y_l|x)} \Big), \quad \text{with } \mathrm{odds}_\theta(y|x) = \tfrac{P_\theta(y|x)}{1 - P_\theta(y|x)}.$$

What matters: there is no π_ref. ORPO trains a single model, in a single pass, without loading the reference policy into memory. On paper it sounds good and in practice it works: a Llama 3 8B aligned with ORPO over 5k pairs takes about 3 hours on 4×H100 and fits in VRAM with aggressive QLoRA on a single RTX 4090.

The λ parameter is the weight of the preference term. Typical: 0.1–0.3. If λ is too high, the model learns to discriminate but forgets the SFT (catastrophic forgetting); if it is too low, the alignment barely shows.

SimPO: do you really need a reference model?

SimPO — Simple Preference Optimization, Meng et al. 2024 — takes ORPO’s question one step further: if ORPO frees itself from π_ref for the combined SFT+preference case, why not free yourself from π_ref in the pure DPO case too?

The SimPO loss:

$$\mathcal{L}_\mathrm{SimPO} = -\log \sigma\!\Big( \tfrac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \tfrac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \Big).$$

Two changes with respect to DPO:

There is no π_ref: absolute log-probs of the trained model are compared directly.
Length-normalization: each log-prob is divided by the length of its answer |y|. This is key because without normalising, long answers tend to have a lower total log-prob (each token contributes its log p < 0), creating an artificial bias.
Explicit margin γ: the loss is low if the difference of normalised log-probs exceeds γ. Typical: γ = 0.5–1.5.

With β = 2.0, γ = 1.0, |y_w| = 120 tokens, |y_l| = 100 tokens and the earlier log-probs:

$$m = \tfrac{2.0}{120} \cdot (-45.2) - \tfrac{2.0}{100} \cdot (-52.1) - 1.0 = -0.753 + 1.042 - 1.0 = -0.711.$$ $$\mathcal{L}_\mathrm{SimPO} = -\log \sigma(-0.711) \approx 1.11.$$

A higher loss than DPO at the same point: SimPO takes longer to converge, but uses half the memory (a single model in VRAM) and removes the dependency on π_ref. It is the dominant option when memory is the bottleneck.

Decision table: which one to use and when

Signal available	Memory available	Prior SFT	Recommended method	Rationale
`(chosen, rejected)` pairs	High (≥ 80 GB GPU)	Yes	DPO	Most established baseline, better reproducibility
`(chosen, rejected)` pairs	Low (24–48 GB GPU)	Yes	SimPO	Removes `π_ref` → ~50 % less VRAM
`(chosen, rejected)` pairs	Any	No	ORPO	SFT and preferences in one pass
Binary 👍/👎 signal without pairs	High	Yes	KTO	The only method native to unpaired data
Binary signal + few pairs	High	Yes	KTO with a DPO sub-batch	Combination documented in TRL 0.13
Multistep trajectories (tool use)	Very high	Yes	Pure RLHF/RFT	Preference-pair methods do not capture the dynamics

Typical dataset magnitudes:

Method	Minimum viable	Sweet spot	Plateau
DPO	1,000 pairs	5,000–20,000	> 50,000
SimPO	2,000 pairs	5,000–20,000	> 50,000
ORPO	3,000 pairs (includes SFT)	10,000–30,000	> 80,000
KTO	5,000 binary examples	20,000–80,000	> 200,000

KTO typically needs 3-4× more volume than DPO because the binary signal is weaker than the comparative one. The trade-off is that the binary signal is the one a production product naturally produces.

The three biases that break the method

All four methods share one problem: they are training on proxies for quality, not on quality. Those proxies have systematic biases the model can exploit trivially.

Length bias

Documented in the original DPO paper and in the literature that followed. Long answers tend to be preferred by humans, probably because they look “more complete”. If the pair dataset inherits that bias, the model learns that lengthening the answer is what gets rewarded, not that better content is what gets rewarded. Result: after 2–3 epochs the model spews waffle.

Mitigations:

DPO: filter the dataset, removing pairs where |y_w| > 1.3 · |y_l| (the 30 % rule).
SimPO: the length-normalization in the loss fixes it by construction.
ORPO / KTO: dataset filtering or an auxiliary length regularisation (DPOP, R-DPO).

The chart shows the usual pattern: with no mitigation, in 3 epochs a Llama 3 8B with DPO can go from answers of about 150 tokens to answers of about 280 tokens, without any improvement in human-evaluated quality. SimPO keeps the length roughly stable.

Position bias (in dataset curation)

If the pairs are generated automatically with an LLM judge (covered in detail in the next post of this series), there is a known bias: judges prefer the first answer they see when the two are comparable. If every pair in the dataset always has the chosen in position A and the rejected in position B, the model does not learn preference: it learns an artefact of the curation process.

Mitigation: random shuffling of the order in the query to the judge and averaging two passes with the order reversed (seen in tools such as Promptfoo and Inspect AI by default).

Distribution shift between the data and `π_ref`

DPO, KTO and SimPO implicitly compare the trained model against a distribution. If the preference data comes from a model (another LLM generating candidates) very different from π_ref, the trained model can explore regions where π_ref has almost zero probability, giving numerically unstable ratios (the log of very small quantities). In practice this shows up as loss explosions, NaN gradients or silent regression.

Mitigation: generate the dataset candidates with π_ref itself whenever possible (rejection sampling over π_ref, with an external judge picking the chosen). That is the canonical on-policy RLHF prescription applied to the offline setting.

Implications on on-premise hardware

The figures below are indicative for a typical May 2026 scenario: Llama 3.1 8B Instruct as π_ref, a dataset of 5,000–20,000 pairs, QLoRA (NF4) with LoRA rank 16 over all the transformer block projectors (q,k,v,o,gate,up,down), effective batch size 16, 1–3 epochs.

On an RTX 4090 (24 GB)

Method	Peak VRAM	Time / epoch (5k pairs)	Notes
DPO	~22 GB	50–80 min	Needs `π_ref` in VRAM even if quantised FP8/INT8
SimPO	~14 GB	45–70 min	No `π_ref`, the natural option on a 4090
ORPO	~16 GB	60–90 min	No `π_ref`, comfortably viable
KTO	~22 GB	90–150 min (10k binary)	Same VRAM as DPO, more data per epoch

The 4090 (24 GB, Ada Lovelace, no NVLink) is perfectly viable for Llama 8B with QLoRA if you pick the method sensibly. For 13B the choice between SimPO/ORPO is no longer a preference, it is a requirement.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Method	Viable model	Time / epoch (10k pairs)	Notes
DPO	Llama 3 70B (4-bit)	60–90 min	Tensor parallel = 4, still comfortable
SimPO	Llama 3 70B (BF16)	50–75 min	Full BF16 fits thanks to having no `π_ref`
ORPO	Llama 3 70B (BF16)	70–100 min	Similar to SimPO in consumption
KTO	Llama 3 70B (4-bit)	100–140 min	Larger datasets offset by parallelism

On an NVLink cluster the operational difference between methods blurs: they all fit. The choice goes back to what kind of signal you have, not to budget.

What we have not covered (upcoming articles)

LoRA and QLoRA fundamentals: the maths of the low-rank adapter that underpins everything above, and why a 70B fits in 24 GB.
LLM-as-judge fundamentals: how to build the judge that generates the chosen/rejected pairs without position or verbosity bias. The next post in this batch covers it.
Online DPO and iterative on-policy: the 2026 research state of the art (Fast-Slow Chasing, RLOO, iterative preference learning) and why it is not production yet.
Distillation and synthetic preference data: when it is worth generating pairs with a large model to train a small one.

References

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization (ICML 2024).
Hong, J., Lee, N., Thorne, J. ORPO: Monolithic Preference Optimization without Reference Model (EMNLP 2024).
Meng, Y., Xia, M., Chen, D. SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
HuggingFace TRL 0.13 — reference implementations: https://huggingface.co/docs/trl.
Tunstall, L. et al. The Alignment Handbook — reproducible recipes: https://github.com/huggingface/alignment-handbook.
Park, R. et al. Disentangling Length from Quality in Direct Preference Optimization (R-DPO, ACL 2024).

Preference-Optimization on lo0 — Tech Blog

Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

TL;DR

You are here: TUNE

The analogy: the sommelier who trains the palate without a theory book

Why DPO exists: Rafailov’s trick

DPO with real numbers

KTO: when you only have 👍/👎

ORPO: SFT and preferences in a single pass

SimPO: do you really need a reference model?

Decision table: which one to use and when

The three biases that break the method

Length bias

Position bias (in dataset curation)

Distribution shift between the data and `π_ref`

Implications on on-premise hardware

On an RTX 4090 (24 GB)

On a generic 4×H100 SXM cluster (320 GB, NVLink)

What we have not covered (upcoming articles)

See also

References

Preference-Optimization on lo0 — Tech Blog

Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

TL;DR

You are here: TUNE

The analogy: the sommelier who trains the palate without a theory book

Why DPO exists: Rafailov’s trick

DPO with real numbers

KTO: when you only have 👍/👎

ORPO: SFT and preferences in a single pass

SimPO: do you really need a reference model?

Decision table: which one to use and when

The three biases that break the method

Length bias

Position bias (in dataset curation)

Distribution shift between the data and π_ref

Implications on on-premise hardware

On an RTX 4090 (24 GB)

On a generic 4×H100 SXM cluster (320 GB, NVLink)

What we have not covered (upcoming articles)

See also

References

Distribution shift between the data and `π_ref`