Dpo on lo0 — Tech Blog

Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

Wed, 27 May 2026 08:00:00 +0200

This post is the natural continuation of continuous fine-tuning in production, which covers the operational cycle (Postgres + SQL queries + hot-swap) that feeds these methods with real data. Here we go inside each one: what each loss optimises, what hypothesis it makes, and why they choose one or the other.

TL;DR

Classic RLHF, the kind in the InstructGPT papers, is practically extinct in production. The reason is not ideological: it is that in 2023 Rafailov and others showed that the reward model does not need to exist as a separate object. With an elegant change of variable, the optimal policy can be trained directly from preference pairs, without going through a reward and without RL. That is DPO. From there the family has branched into four methods that coexist in 2026: DPO when you have (chosen, rejected) pairs, KTO when you only have a binary 👍/👎 signal, ORPO when you want SFT and preferences in the same pass to save memory, and SimPO when you also want to get rid of the reference model and normalise by length. This post explains exactly what each loss does, proves it with an end-to-end numerical example, gives the real decision table between the four and takes apart the three biases that kill the method in production when nobody watches them.

You are here: TUNE

The six-stage LLMOps pipeline places Tune between Data and Eval. Inside Tune three modalities coexist: SFT (supervised fine-tuning), preference optimization (what this post covers) and agent training / RFT. What follows is the inside of the second modality.

The analogy: the sommelier who trains the palate without a theory book

Imagine you want to train a sommelier. You have two routes. The first is teaching them oenological theory: grape varieties, terroirs, vinification methods, barrel types. You give them cards with canonical wine descriptions and ask them to memorise them. That is SFT: (prompt, ideal answer) pairs. It works as general education, but the sommelier who comes out of it cannot tell two excellent Riojas apart.

The second route is the blind comparator. You put two identical opaque glasses in front of them. You say “this one is better than this one”. You do not explain why. You repeat the exercise a thousand times, with a thousand different pairs. After a while, the sommelier does not need you to say it: they have a trained palate. They have not learned oenological theory, they have learned to discriminate.

DPO, KTO, ORPO and SimPO are four variants of the second route. All four train the model to discriminate, not to memorise. The differences between them are:

DPO: two glasses at each tasting, the sommelier always knows which one is the “good” one and which is the “bad” one.
KTO: a single glass each time, they are only told “you would like this” or “you would not like this”. No pairs.
ORPO: the tasting also includes a small embedded theory class (SFT in parallel) so the catalogue is not forgotten.
SimPO: like DPO, but the sommelier does not compare against “what your master would have said” (reference model); they compare the two glasses directly, normalising by the amount of liquid in each.

The analogy is not decorative: the rest of the post is that same idea expressed mathematically.

Why DPO exists: Rafailov’s trick

To understand DPO it helps to walk through in thirty seconds what it replaces. Classic RLHF, the InstructGPT kind, has three phases:

Phase 1 – SFT. You train the model on (prompt, ideal answer). Out comes π_ref: the reference policy. It is the “educated” model.

Phase 2 – Reward model. On the same model (a different head), you train a regressor: given (prompt, chosen, rejected), it learns to give a higher score to the chosen. Out comes r_φ(x, y).

Phase 3 – PPO. You take π_ref as the starting point and train another copy π_θ so that it maximises r_φ(x, π_θ(x)) with a KL penalty to keep it from drifting too far from π_ref. That requires generating rollouts (on-policy decode at every step), keeping the three models in memory (π_θ, π_ref, r_φ), and a classic RL setup: unstable, sensitive to hyperparameters and famous for its reproducibility problems.

Rafailov’s observation was: phase 3 has a closed-form solution. If you write out the exact optimisation problem PPO solves

$$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D, \, y \sim \pi_\theta(\cdot|x)} \big[ r_\phi(x,y) \big] - \beta \, \mathrm{KL}\big(\pi_\theta(\cdot|x) \,\|\, \pi_\mathrm{ref}(\cdot|x)\big)$$

it turns out the optimal policy has the form

$$\pi^{*}(y|x) = \frac{1}{Z(x)} \, \pi_\mathrm{ref}(y|x) \, \exp\!\left( \tfrac{1}{\beta} r_\phi(x,y) \right)$$

and from there you can solve for the implicit reward:

$$r_\phi(x,y) = \beta \log \frac{\pi^{*}(y|x)}{\pi_\mathrm{ref}(y|x)} + \beta \log Z(x).$$

The second term is a function of x only and cancels out when you compare two answers to the same prompt. The reward does not need to be learned: it is implicit in the log-prob ratio between the trained model and the reference one. If you plug that into the Bradley-Terry model for preferences (the formula that says “the probability that yw is preferred to yl is σ(r(x,yw) - r(x,yl))”), out comes the DPO loss:

$$\mathcal{L}_\mathrm{DPO}(\pi_\theta;\pi_\mathrm{ref}) = -\,\mathbb{E}_{(x, y_w, y_l) \sim D} \log \sigma\!\Big( \beta \big[ \log \tfrac{\pi_\theta(y_w|x)}{\pi_\mathrm{ref}(y_w|x)} - \log \tfrac{\pi_\theta(y_l|x)}{\pi_\mathrm{ref}(y_l|x)} \big] \Big).$$

That is the whole of DPO. There is no reward model, no RL, no rollouts. Only log-probs over static data.

DPO with real numbers

The formula is less intimidating when you evaluate it with an example. Imagine a pair from the dataset:

x = “Explain what a KVM switch is”.
y_w = correct answer (chosen).
y_l = confusing answer (rejected).

After a forward pass we have four numbers (sums of per-token log-probs, negative sign because each log p_token ≤ 0):

Quantity	Value
`log π_θ(y_w \| x)`	−45.2
`log π_θ(y_l \| x)`	−52.1
`log π_ref(y_w \| x)`	−47.3
`log π_ref(y_l \| x)`	−50.8

The “ratios” are the improvement of the trained model over the reference one for each answer:

For y_w: −45.2 − (−47.3) = +2.1 → the trained model gives it more probability than the reference one. Good.
For y_l: −52.1 − (−50.8) = −1.3 → the trained model gives it less probability. Also good.

With β = 0.1 (a typical value) the “margin” inside the logarithm is:

$$m = \beta \cdot (2.1 - (-1.3)) = 0.1 \cdot 3.4 = 0.34$$

And the loss:

$$\mathcal{L}_\mathrm{DPO} = -\log \sigma(0.34) = -\log(0.584) \approx 0.538.$$

The intuition is visible straight away: the more the chosen log-prob rises and the more the rejected one falls (relative to π_ref), the more positive m becomes, the higher the sigmoid and the lower the loss. If the margin is negative (the model gets it wrong), σ(m) < 0.5 and the loss blows up. The gradient pushes the model to increase π_θ(y_w|x) and lower π_θ(y_l|x).

The role of β: it is the temperature of the alignment. If β is small, the model is allowed to drift a long way from π_ref; if it is large, the KL weighs heavily and the model barely moves. Typical value in 2026: 0.05–0.3, with 0.1 as a starting point.

KTO: when you only have 👍/👎

DPO needs pairs. In practice, that is almost never what the product gives you: what the product gives you is binary feedback (a thumbs up or a thumbs down, a conversion or an abandonment). KTO — Kahneman-Tversky Optimization, Ethayarajh et al. 2024 — solves exactly that case.

The intuition comes from Kahneman and Tversky’s prospect theory: humans are more sensitive to losses than to gains of the same magnitude (loss aversion: losing 100 € hurts more than gaining 100 €). KTO carries that over into the loss:

$$\mathcal{L}_\mathrm{KTO}(x, y) = \begin{cases} \lambda_d \big[ 1 - \sigma\!\big(\beta \cdot ( h_\theta(x,y) - z_0 ) \big) \big] & \text{if } y \text{ is desirable},\\ \lambda_u \big[ 1 - \sigma\!\big(\beta \cdot ( z_0 - h_\theta(x,y) ) \big) \big] & \text{if } y \text{ is undesirable}, \end{cases}$$

where h_θ(x,y) = log(π_θ(y|x) / π_ref(y|x)) is exactly the same ratio that appeared in DPO, z_0 is an estimate of the batch KL divergence (it acts as a “neutral point”) and λ_d, λ_u are the desirable / undesirable weights.

The critical point: KTO does not need pairs. Each example is (prompt, answer, binary label). This fits real product telemetry. The usual practical rule is λ_u > λ_d (e.g. 1.0 vs 0.33) when the database has more 👍 than 👎, so that the negative signal is not diluted.

KTO works particularly well in two scenarios:

Products with explicit feedback UX (chatbots with 👍/👎): each thumbs is a direct KTO example, no need to synthesise pairs.
Imbalanced datasets (far more 👍 than 👎, or the other way round): the weights λ_d, λ_u handle it explicitly.

ORPO: SFT and preferences in a single pass

DPO assumes you have already done SFT. You train two phases: first SFT to get π_ref, then DPO on top of π_ref. Two passes over the data, two optimisations, two models in memory.

ORPO — Odds Ratio Preference Optimization, Hong et al. 2024 — merges both phases. The loss combines two terms:

$$\mathcal{L}_\mathrm{ORPO} = \mathcal{L}_\mathrm{SFT}(y_w) + \lambda \cdot \mathcal{L}_\mathrm{OR}(y_w, y_l)$$

The first term is classic SFT on the chosen answer (negative cross-entropy). The second is the odds ratio between chosen and rejected:

$$\mathcal{L}_\mathrm{OR} = -\log \sigma\!\Big( \log \tfrac{\mathrm{odds}_\theta(y_w|x)}{\mathrm{odds}_\theta(y_l|x)} \Big), \quad \text{with } \mathrm{odds}_\theta(y|x) = \tfrac{P_\theta(y|x)}{1 - P_\theta(y|x)}.$$

What matters: there is no π_ref. ORPO trains a single model, in a single pass, without loading the reference policy into memory. On paper it sounds good and in practice it works: a Llama 3 8B aligned with ORPO over 5k pairs takes about 3 hours on 4×H100 and fits in VRAM with aggressive QLoRA on a single RTX 4090.

The λ parameter is the weight of the preference term. Typical: 0.1–0.3. If λ is too high, the model learns to discriminate but forgets the SFT (catastrophic forgetting); if it is too low, the alignment barely shows.

SimPO: do you really need a reference model?

SimPO — Simple Preference Optimization, Meng et al. 2024 — takes ORPO’s question one step further: if ORPO frees itself from π_ref for the combined SFT+preference case, why not free yourself from π_ref in the pure DPO case too?

The SimPO loss:

$$\mathcal{L}_\mathrm{SimPO} = -\log \sigma\!\Big( \tfrac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \tfrac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \Big).$$

Two changes with respect to DPO:

There is no π_ref: absolute log-probs of the trained model are compared directly.
Length-normalization: each log-prob is divided by the length of its answer |y|. This is key because without normalising, long answers tend to have a lower total log-prob (each token contributes its log p < 0), creating an artificial bias.
Explicit margin γ: the loss is low if the difference of normalised log-probs exceeds γ. Typical: γ = 0.5–1.5.

With β = 2.0, γ = 1.0, |y_w| = 120 tokens, |y_l| = 100 tokens and the earlier log-probs:

$$m = \tfrac{2.0}{120} \cdot (-45.2) - \tfrac{2.0}{100} \cdot (-52.1) - 1.0 = -0.753 + 1.042 - 1.0 = -0.711.$$ $$\mathcal{L}_\mathrm{SimPO} = -\log \sigma(-0.711) \approx 1.11.$$

A higher loss than DPO at the same point: SimPO takes longer to converge, but uses half the memory (a single model in VRAM) and removes the dependency on π_ref. It is the dominant option when memory is the bottleneck.

Decision table: which one to use and when

Signal available	Memory available	Prior SFT	Recommended method	Rationale
`(chosen, rejected)` pairs	High (≥ 80 GB GPU)	Yes	DPO	Most established baseline, better reproducibility
`(chosen, rejected)` pairs	Low (24–48 GB GPU)	Yes	SimPO	Removes `π_ref` → ~50 % less VRAM
`(chosen, rejected)` pairs	Any	No	ORPO	SFT and preferences in one pass
Binary 👍/👎 signal without pairs	High	Yes	KTO	The only method native to unpaired data
Binary signal + few pairs	High	Yes	KTO with a DPO sub-batch	Combination documented in TRL 0.13
Multistep trajectories (tool use)	Very high	Yes	Pure RLHF/RFT	Preference-pair methods do not capture the dynamics

Typical dataset magnitudes:

Method	Minimum viable	Sweet spot	Plateau
DPO	1,000 pairs	5,000–20,000	> 50,000
SimPO	2,000 pairs	5,000–20,000	> 50,000
ORPO	3,000 pairs (includes SFT)	10,000–30,000	> 80,000
KTO	5,000 binary examples	20,000–80,000	> 200,000

KTO typically needs 3-4× more volume than DPO because the binary signal is weaker than the comparative one. The trade-off is that the binary signal is the one a production product naturally produces.

The three biases that break the method

All four methods share one problem: they are training on proxies for quality, not on quality. Those proxies have systematic biases the model can exploit trivially.

Length bias

Documented in the original DPO paper and in the literature that followed. Long answers tend to be preferred by humans, probably because they look “more complete”. If the pair dataset inherits that bias, the model learns that lengthening the answer is what gets rewarded, not that better content is what gets rewarded. Result: after 2–3 epochs the model spews waffle.

Mitigations:

DPO: filter the dataset, removing pairs where |y_w| > 1.3 · |y_l| (the 30 % rule).
SimPO: the length-normalization in the loss fixes it by construction.
ORPO / KTO: dataset filtering or an auxiliary length regularisation (DPOP, R-DPO).

The chart shows the usual pattern: with no mitigation, in 3 epochs a Llama 3 8B with DPO can go from answers of about 150 tokens to answers of about 280 tokens, without any improvement in human-evaluated quality. SimPO keeps the length roughly stable.

Position bias (in dataset curation)

If the pairs are generated automatically with an LLM judge (covered in detail in the next post of this series), there is a known bias: judges prefer the first answer they see when the two are comparable. If every pair in the dataset always has the chosen in position A and the rejected in position B, the model does not learn preference: it learns an artefact of the curation process.

Mitigation: random shuffling of the order in the query to the judge and averaging two passes with the order reversed (seen in tools such as Promptfoo and Inspect AI by default).

Distribution shift between the data and `π_ref`

DPO, KTO and SimPO implicitly compare the trained model against a distribution. If the preference data comes from a model (another LLM generating candidates) very different from π_ref, the trained model can explore regions where π_ref has almost zero probability, giving numerically unstable ratios (the log of very small quantities). In practice this shows up as loss explosions, NaN gradients or silent regression.

Mitigation: generate the dataset candidates with π_ref itself whenever possible (rejection sampling over π_ref, with an external judge picking the chosen). That is the canonical on-policy RLHF prescription applied to the offline setting.

Implications on on-premise hardware

The figures below are indicative for a typical May 2026 scenario: Llama 3.1 8B Instruct as π_ref, a dataset of 5,000–20,000 pairs, QLoRA (NF4) with LoRA rank 16 over all the transformer block projectors (q,k,v,o,gate,up,down), effective batch size 16, 1–3 epochs.

On an RTX 4090 (24 GB)

Method	Peak VRAM	Time / epoch (5k pairs)	Notes
DPO	~22 GB	50–80 min	Needs `π_ref` in VRAM even if quantised FP8/INT8
SimPO	~14 GB	45–70 min	No `π_ref`, the natural option on a 4090
ORPO	~16 GB	60–90 min	No `π_ref`, comfortably viable
KTO	~22 GB	90–150 min (10k binary)	Same VRAM as DPO, more data per epoch

The 4090 (24 GB, Ada Lovelace, no NVLink) is perfectly viable for Llama 8B with QLoRA if you pick the method sensibly. For 13B the choice between SimPO/ORPO is no longer a preference, it is a requirement.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Method	Viable model	Time / epoch (10k pairs)	Notes
DPO	Llama 3 70B (4-bit)	60–90 min	Tensor parallel = 4, still comfortable
SimPO	Llama 3 70B (BF16)	50–75 min	Full BF16 fits thanks to having no `π_ref`
ORPO	Llama 3 70B (BF16)	70–100 min	Similar to SimPO in consumption
KTO	Llama 3 70B (4-bit)	100–140 min	Larger datasets offset by parallelism

On an NVLink cluster the operational difference between methods blurs: they all fit. The choice goes back to what kind of signal you have, not to budget.

What we have not covered (upcoming articles)

LoRA and QLoRA fundamentals: the maths of the low-rank adapter that underpins everything above, and why a 70B fits in 24 GB.
LLM-as-judge fundamentals: how to build the judge that generates the chosen/rejected pairs without position or verbosity bias. The next post in this batch covers it.
Online DPO and iterative on-policy: the 2026 research state of the art (Fast-Slow Chasing, RLOO, iterative preference learning) and why it is not production yet.
Distillation and synthetic preference data: when it is worth generating pairs with a large model to train a small one.

References

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization (ICML 2024).
Hong, J., Lee, N., Thorne, J. ORPO: Monolithic Preference Optimization without Reference Model (EMNLP 2024).
Meng, Y., Xia, M., Chen, D. SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
HuggingFace TRL 0.13 — reference implementations: https://huggingface.co/docs/trl.
Tunstall, L. et al. The Alignment Handbook — reproducible recipes: https://github.com/huggingface/alignment-handbook.
Park, R. et al. Disentangling Length from Quality in Direct Preference Optimization (R-DPO, ACL 2024).

Continuous fine-tuning in production: from real traffic to the deployed adapter

Thu, 21 May 2026 10:00:00 +0200

TL;DR

Continuous fine-tuning is not “training the model every so often”. It is a closed loop where real production traffic generates the datasets, a short pipeline trains a LoRA adapter, a battery of evaluations decides whether it is promoted, and vLLM loads it without restarting. The state of the art in May 2026 has fragmented the stack: it is no longer DPO against everything, but a choice between SFT, DPO, KTO, ORPO and SimPO according to the type of signal your product captures. What has consolidated the pattern is the combination of PostgreSQL 18 + pgvector 0.8 as the nervous system of the pipeline, traffic capture, dataset versioning, eval results, adapter registry, together with vLLM multi-LoRA hot-swap, which turns deployment into an HTTP call. This article takes the cycle apart with concrete schemas, real queries, and the numbers it costs on an RTX 4090 against a 4×H100 cluster.

You are here: Tune + Retrain

This post crosses two stages of the six-stage LLMOps pipeline: the decision to train a new adapter (the Tune stage) is triggered by the signals from Observe that travel through the Retrain stage until the loop closes. The post takes apart the complete circuit between the two boxes.

Picture a neighbourhood restaurant with a signature dish that works, but the chef knows it can be tuned. Every night things happen:

Some diners leave part of the dish: a weak signal that something did not quite land.
Others ask for another version (“could you put less salt in it?”): an explicit, directional signal.
Others finish the dish and come back the following week: the only signal that really matters, but it arrives late.
And a select group offers an opinion without being asked, usually a negative one.

The chef does not redo the menu every night. He does something more interesting: he writes down in a notebook the dishes served, the returns, the changes requested, the tips. Every so often, he reads the whole notebook, decides on minimal adjustments to a recipe, tries the new version at a private table with his staff, and only if they take to it does he add it to the next day’s menu. Sometimes he even serves two different versions of the dish to different tables for a week, measures what happens, and picks one.

That is continuous fine-tuning. The notebook is Postgres. The dish is the base model. The notes are feedback signals, explicit and implicit. The “minimal adjustment” is a 30 MB LoRA adapter. The private table is the battery of automated evaluations. The next day’s menu is vLLM with multi-LoRA hot-swap, which loads the new adapter without restarting the service. Serving two versions to different tables is A/B testing with real traffic.

The analogy is exact on one critical point: the chef does not throw away the original recipe. He keeps the base recipe and holds a separate notebook with the “modifications that work well for the neighbourhood regulars”. That notebook is the LoRA adapter: on top of the base model, not in its place.

The cycle, taken apart

Before going into components, it helps to fix the complete flow. These seven steps are what any serious team replicates with variations:

The cycle lasts between 1 and 4 weeks in real production. What changes between teams is the tempo (faster in assistant chat, slower in regulated banking) and the details of each step. The structure is the same.

Why continuous fine-tuning (and why it is not RAG)

Before going deeper, a distinction that people still confuse. Fine-tuning is for form, not for facts. If your problem is that the model does not know the customer’s rates or the updated catalogue, do not fine-tune: use RAG. If your problem is that the model responds in a tone that does not fit, does not respect your JSON format, refuses legitimate cases or invents structure, then fine-tuning is the answer.

In 2026 the boundary is well established by community practice:

Observed problem	Solution
The model does not know X (X changes weekly)	RAG
The model knows X but responds badly in tone or format	SFT fine-tuning
There are two ways of responding and I prefer one over the other	Fine-tuning with preferences (DPO/KTO/ORPO/SimPO)
The model reasons badly in a verifiable domain (code, maths)	RL with verifiable reward (GRPO/DAPO)
The model is competent, it just needs a memory of facts	RAG, not fine-tuning

Continuous fine-tuning is the disciplined version of the second and third cases. The key word is continuous: it is not a one-off “we aligned the model” event, it is a process that runs every time the traffic distribution drifts far enough, or new use cases appear.

The four techniques according to the signal you capture

The most important change of the last 12 months has been the end of DPO’s monopoly. In 2024 every team doing alignment used DPO with (chosen, rejected) pairs. In 2026 the choice is finer and depends on what the signal you collect in your product looks like:

Real signal in the product	Recommended technique	Why
Labelled correct examples (input → expected output)	SFT + LoRA	Still the baseline. 500-5,000 examples are enough for style.
Explicit `(chosen, rejected)` pairs	DPO or SimPO	SimPO removes the reference model → 50 % less VRAM in training.
Loose 👍 / 👎 on responses	KTO	The method that fits real telemetry most naturally.
SFT and preferences in a single pass	ORPO	A single model in memory, avoids drift between phases.
Verifiable reward (tests, solutions)	GRPO / DAPO	Reasoning, not chat. A different world.

The practical rule: design the feedback capture in the product thinking about which method you will be able to use afterwards. If your UI only has 👍/👎, you force the path to KTO. If you add a “regenerate response” button, you unlock DPO from regenerate-as-rejected (we will see it below). If you add an “edit response” button, the edited response becomes high-quality direct SFT.

There is a cost detail that gets little publicity. DPO needs to keep two models in memory: the one you are training and the reference one. SimPO removes that second model. ORPO does too. For a Llama 3 8B in BF16 this is the difference between needing ~32 GB of active VRAM during training (DPO) or ~16 GB (SimPO/ORPO). It is the difference between the training fitting on an RTX 4090 with aggressive QLoRA, or not fitting without offload.

Postgres as the nervous system of the pipeline

Here is the strong technical opinion of this article, and it is one worth defending with data: Postgres 18 + pgvector 0.8 + an S3/MinIO bucket for the weights is enough for the whole pipeline. You do not need MLflow, you do not need lakeFS, you do not need DVC.

This is not ideological minimalism. It is about three concrete advantages that no alternative stack matches in the on-premise scenario with compliance:

A single source of truth, a single authorisation model. The ACLs you already have for Postgres cover the training data, the eval results, the adapter registry and the audit log. You do not multiply control planes.
SQL as the pipeline’s universal language. The query that generates the dataset, the eval gate predicate, the A/B traffic assignment, the promotion decision: it is all SQL. Your team already knows SQL.
Audit and cryptographic reproducibility for free. The pg_audit and pgcrypto extensions, combined with set_hash over the dataset, give you cryptographic traceability with no additional code. It is a subject that would fill an article of its own.

A concrete schema

We start with the traffic table, partitioned by week so that DROP PARTITION is cheap:

CREATE TABLE obs.inference_log (
 id BIGSERIAL,
 request_id UUID NOT NULL,
 tenant_id INT NOT NULL,
 user_hash BYTEA, -- GDPR pseudonymisation
 adapter_id TEXT NOT NULL, -- e.g. "support-es-v4.1"
 experiment TEXT, -- e.g. "rerank-v2-canary"
 variant CHAR(1), -- 'A' | 'B' | NULL
 messages JSONB NOT NULL,
 completion TEXT,
 ttft_ms INT,
 tokens_in INT,
 tokens_out INT,
 -- feedback signals
 fb_explicit SMALLINT, -- -1/0/+1 (KTO-ready)
 fb_regen BOOLEAN DEFAULT false, -- user regenerated -> DPO-rejected
 fb_edited BOOLEAN DEFAULT false, -- user edited -> SFT golden
 parent_id BIGINT, -- self-reference for regenerate
 -- vector and meta
 embedding HALFVEC(1024), -- pgvector 0.8, half the RAM
 pii_flags SMALLINT DEFAULT 0, -- bitmask
 created_at TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);

CREATE TABLE obs.inference_log_2026w21 PARTITION OF obs.inference_log
 FOR VALUES FROM ('2026-05-18') TO ('2026-05-25');

CREATE INDEX ON obs.inference_log_2026w21
 USING hnsw (embedding halfvec_cosine_ops);
CREATE INDEX ON obs.inference_log_2026w21
 (tenant_id, adapter_id, created_at);

Three decisions deserve a note:

HALFVEC(1024). Native FP16 vectors in pgvector 0.8. Half the RAM and disk, with a loss of precision that is irrelevant for semantic deduplication. This alone, at a scale of millions of rows, saves between 4 and 8 GB.

Weekly partitioning by time range. At 90 days, DROP TABLE obs.inference_log_2026wXX frees space in milliseconds without a prolonged lock. Autovacuum never touches frozen partitions again.

Self-referencing parent_id. The user regenerates the response → a new row is inserted with parent_id pointing at the previous one. That will give us a DPO dataset without touching the UX.

The adapter registry

CREATE TABLE serve.adapter (
 id TEXT PRIMARY KEY,
 base_model TEXT NOT NULL,
 rank INT, alpha INT,
 target_modules JSONB,
 method TEXT, -- 'sft'|'dpo'|'kto'|'orpo'|'simpo'
 training_run_id UUID,
 dataset_snapshot_id UUID,
 weights_uri TEXT, -- s3://.../v4.2.safetensors
 eval_summary JSONB,
 status TEXT NOT NULL, -- 'training'|'canary'|'prod'|'retired'
 traffic_pct NUMERIC(5,2) DEFAULT 0,
 promoted_at TIMESTAMPTZ
);

The vLLM router reads this table with a TTL of a few seconds. An UPDATE serve.adapter SET status='prod', traffic_pct=100 WHERE id='v4.2' is a promotion. An UPDATE ... SET status='retired' is a rollback. The audit of who did what and when comes from pg_audit without writing a single additional line of code.

Generating DPO and KTO datasets from real traffic

This is where the elegance of the schema pays off. The dataset is not a static file: it is a materialised view built with SQL over obs.inference_log.

KTO dataset from 👍/👎

KTO is the method that best fits the signal any decent chat product captures. The query:

CREATE MATERIALIZED VIEW train.kto_v3_candidate AS
SELECT
 messages AS prompt,
 completion AS response,
 CASE WHEN fb_explicit > 0 THEN true ELSE false END AS label,
 adapter_id, created_at
FROM obs.inference_log
WHERE fb_explicit != 0
 AND created_at > now() - interval '60 days'
 AND pii_flags = 0
 AND tenant_id IN (SELECT id FROM tenant WHERE consent_training);

Simple. Every row with explicit feedback becomes an example (prompt, response, desirable_yes_no). KTO trains directly on this signal, with no need to build pairs.

DPO dataset from “regenerate”

The trick that is worth this article on its own. When the user presses “regenerate response”, they are giving an extraordinarily strong signal: the first response was no good to them. If the second is neither regenerated nor rated negatively, we assume it was. That is a DPO pair without a single extra click in the UI:

CREATE MATERIALIZED VIEW train.dpo_v3_candidate AS
SELECT
 rej.messages AS prompt,
 cho.completion AS chosen,
 rej.completion AS rejected,
 rej.adapter_id
FROM obs.inference_log rej
JOIN obs.inference_log cho ON cho.parent_id = rej.id
WHERE rej.fb_regen = true
 AND cho.fb_explicit >= 0
 -- length bias mitigation in classic DPO
 AND cho.tokens_out BETWEEN rej.tokens_out * 0.5
 AND rej.tokens_out * 2.5;

The clause on lengths is the cheap cure for the length bias documented in DPO. Without it, the model learns that “longer = better” because the responses the user accepts tend to be slightly longer. With SimPO or ORPO this filter is optional; with classic DPO it is necessary.

Semantic deduplication with pgvector

Before training, dedup. Two almost identical prompts in the dataset are noise that biases the model:

WITH ranked AS (
 SELECT id, embedding,
 row_number() OVER (
 PARTITION BY hashtext(messages::text)
 ORDER BY fb_explicit DESC, created_at DESC
 ) AS rn
 FROM obs.inference_log
 WHERE created_at > now() - interval '60 days'
)
DELETE FROM train.kto_v3_candidate kto
USING ranked r
WHERE r.rn > 1 AND kto.id = r.id;

And for the semantic duplicates (paraphrases) we use pgvector 0.8 directly with an iterative index scan:

-- Find near-duplicates of any given example
SELECT id, messages, embedding <=> $1 AS dist
FROM obs.inference_log
WHERE created_at > now() - interval '60 days'
 AND embedding <=> $1 < 0.05
ORDER BY embedding <=> $1
LIMIT 50;

The iterative scan is a key improvement in pgvector 0.8: before, the HNSW index could return fewer results than requested when there were additional filters (WHERE); now it iterates until it meets the limit. Without that improvement, curation queries over datasets of millions of rows were unworkable without a brutal pre-filter.

Eval gates: three stages, all SQL

The most common mistake when implementing continuous fine-tuning is skipping or watering down the eval gates. That turns the cycle into a roulette wheel. The pattern that works in 2026 is three stages, each with a different latency/coverage trade-off:

And this is where Postgres shines again: the promotion gate is expressed as an SQL predicate. Nothing more:

CREATE TABLE train.eval_result (
 adapter_id TEXT REFERENCES serve.adapter(id),
 suite_id TEXT, -- 'safety-es', 'support-helpfulness'
 metric TEXT,
 score NUMERIC,
 judge_model TEXT,
 judged_at TIMESTAMPTZ DEFAULT now(),
 PRIMARY KEY (adapter_id, suite_id, metric)
);

CREATE OR REPLACE FUNCTION serve.can_promote(candidate TEXT, current TEXT)
RETURNS BOOLEAN AS $$
 SELECT NOT EXISTS (
 SELECT 1
 FROM train.eval_result c
 JOIN train.eval_result p USING (suite_id, metric)
 WHERE c.adapter_id = candidate
 AND p.adapter_id = current
 AND suite_id IN ('safety-es','support-helpfulness','refusal-rate')
 AND c.score < p.score * 0.98 -- 2 % tolerance
 );
$$ LANGUAGE sql STABLE;

An SQL function as the gate. Usable from CI with psql -c "SELECT serve.can_promote('v4.2','v4.1')" and an exit code of 0/1. No orchestrator needed, no specific UI needed. The audit stays in the Postgres log.

vLLM multi-LoRA: the deploy is an HTTP POST

Two years ago, deploying a new fine-tune meant rotating inference pods. Today it is an HTTP call. vLLM 0.7+ supports loading and unloading LoRA adapters hot, keeping several resident in VRAM and choosing the right one per request.

Server configuration:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --enable-lora \
 --max-loras 4 \
 --max-lora-rank 64 \
 --env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Deploying a new adapter:

curl -X POST http://localhost:8000/v1/load_lora_adapter \
 -H "Content-Type: application/json" \
 -d '{
 "lora_name": "support-es-v4.2",
 "lora_path": "/mnt/adapters/support-es-v4.2"
 }'

From that moment on, requests including "model": "support-es-v4.2" are served with that adapter applied on top of the base model. Switching between adapters has negligible latency (the most recent research on Activated LoRA takes this to levels where the switching cost is invisible).

This changes operations substantially. Deploying a new fine-tune stops being an infrastructure event and becomes a state change in Postgres. The router queries the serve.adapter table, sees that v4.2 is in canary with traffic_pct=5, and directs 5 % of requests to the new adapter. The exact route of that 5 % is decided with deterministic hashing of the user_id so that the same user always sees the same variant (sticky):

-- No assignment table, no additional state
-- the variant is computed in-place in SQL or in the router:
SELECT
 CASE WHEN hashtext($user_id || $experiment) % 100
 < (SELECT traffic_pct FROM serve.adapter WHERE id = $candidate)
 THEN $candidate ELSE $current END AS adapter_id;

A/B with real traffic: measure or live deceived

The eval gates measure against fixed benchmarks. That is necessary but insufficient. Reality is only measured with real traffic. Once the adapter is in canary, what matters are the online metrics measured over obs.inference_log for each variant:

SELECT
 adapter_id,
 COUNT(*) AS n,
 AVG(fb_explicit) AS mean_score,
 STDDEV(fb_explicit) / SQRT(COUNT(*)) AS sem,
 AVG(ttft_ms) AS ttft_avg,
 percentile_cont(0.5) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p50,
 percentile_cont(0.95) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p95,
 AVG(CASE WHEN fb_regen THEN 1 ELSE 0 END) AS regen_rate
FROM obs.inference_log
WHERE experiment = $1
 AND created_at > now() - interval '7 days'
GROUP BY adapter_id;

What to look at: explicit feedback, latency (TTFT, p50, p95), regeneration rate. An adapter that raises the mean feedback but also raises the regeneration rate is suspicious, probably it is responding in a flashier but less useful way. An adapter that lowers latency but lowers feedback may be worth studying: it may be being more concise than it should.

Promotion to prod happens when, after 24-72 hours in canary, the candidate adapter beats the current one on at least one key metric without degrading the others. Again: it is an UPDATE in Postgres.

Applied to typical on-premise hardware

Let us come down to two representative configurations, one for iteration and one for production.

Case 1 — RTX 4090 (24 GB) for development iteration

An RTX 4090 with 4-bit QLoRA can train adapters on an 8B model without trouble. The VRAM budget combines four components; the KV cache during the intermediate evaluations is not negligible and deserves an explicit margin:

8B base model in 4-bit: ~5 GB
Activations + gradients: ~8 GB (depends on batch and context)
Optimizer state (LoRA r=16): ~0.5 GB
KV cache during eval: ~2 GB
Safety margin: ~8 GB

Typical times (an estimate based on community benchmarks; measure in the lab before promising anything):

Dataset	Technique	Adapter rank	Approximate time
1,000 SFT examples	LoRA r=16	16	20-40 min
5,000 SFT examples	LoRA r=32	32	2-4 h
2,000 DPO pairs	LoRA r=16	16	1-2 h
5,000 KTO examples	LoRA r=32	32	3-5 h

This puts the iteration cycle, a change in the dataset, retrain, eval, look at the number, in the range of a working day. Enough to validate a hypothesis before moving anything to the production cluster.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink) for production

With a cluster of this order the whole picture changes. You can:

Train LoRA on a 70B in BF16 with no quantisation with tensor parallel = 4.
Do full DPO with a resident reference model when the reference is quantised to FP8, or move to SimPO / ORPO, which remove that intermediate model and simplify VRAM planning (see the table of techniques above).
Support multi-tenant fine-tuning: several customer adapters training in parallel in separate logical pipelines, each isolated in a different Postgres partition with its own ACLs.
Serve multi-LoRA with --max-loras 8 on top of the base model without concurrency lowering throughput perceptibly.

The practical budgeting rule: over a 12-month horizon, a team with this cluster can run ~150-200 continuous fine-tuning cycles (training + eval + canary + promotion or discard) if the discipline around the dataset and the eval gates is strict. If it is not, it will run twice as many but with half the usefulness.

Position within the architecture: what this article covers and what it does not

To place the scope: the cycle drawn at the start has seven boxes, all of them covered here in their mechanics. Three cross-cutting layers are deliberately left out, and they are the ones that end up separating a pipeline that works technically from one that survives an audit:

Cryptographic provenance and traceability. We have mentioned dataset_snapshot and pg_audit, but the complete mechanics, the set_hash over the examples, the integration with the EU AI Act, the frozen query_sql as proof of what trained the model, is enough for a whole analysis.
Judge calibration. We have assumed LLM-as-judge works. It needs calibrating against a human rubric on at least 100 cases per critical suite before trusting it. Without that calibration, the eval gates are theatre.
The forgetting problem. What happens if a user exercises their GDPR right to be forgotten and their interactions were part of the dataset of an adapter already in production? There is no clean solution. There are options, incremental retrain, sample-level machine unlearning, negative LoRA, and it is worth knowing them before a customer asks.

What we have not covered (upcoming articles)

Cryptographic provenance on Postgres: how set_hash and a frozen query_sql compose an auditable chain of custody under the EU AI Act.
Honest judge calibration: why score > 0.85 means nothing without a human baseline, and how to build that baseline without it costing a month of work.
The forgetting problem in adapters: sample-level machine unlearning, incremental retrain and other techniques for answering GDPR without throwing the adapter away.
Online DPO and on-policy continual learning: the state of 2026 research (Fast-Slow Chasing, RLOO, iterative on-policy) and why it is not production yet.

References

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022).
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023).
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
Hong et al., ORPO: Monolithic Preference Optimization without Reference Model (2024).
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM.
Official vLLM Multi-LoRA documentation: https://docs.vllm.ai/en/stable/features/lora/.
Official pgvector 0.8 documentation: https://github.com/pgvector/pgvector.
TRL (HuggingFace) docs: https://huggingface.co/docs/trl.
EU AI Act, consolidated text and application timeline: https://artificialintelligenceact.eu/.

Dpo on lo0 — Tech Blog

Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

TL;DR

You are here: TUNE

The analogy: the sommelier who trains the palate without a theory book

Why DPO exists: Rafailov’s trick

DPO with real numbers

KTO: when you only have 👍/👎

ORPO: SFT and preferences in a single pass

SimPO: do you really need a reference model?

Decision table: which one to use and when

The three biases that break the method

Length bias

Position bias (in dataset curation)

Distribution shift between the data and π_ref

Implications on on-premise hardware

On an RTX 4090 (24 GB)

On a generic 4×H100 SXM cluster (320 GB, NVLink)

What we have not covered (upcoming articles)

See also

References

Continuous fine-tuning in production: from real traffic to the deployed adapter

TL;DR

You are here: Tune + Retrain

The analogy: the restaurant that tunes its menu

The cycle, taken apart

Why continuous fine-tuning (and why it is not RAG)

The four techniques according to the signal you capture

Postgres as the nervous system of the pipeline

A concrete schema

The adapter registry

Generating DPO and KTO datasets from real traffic

KTO dataset from 👍/👎

DPO dataset from “regenerate”

Semantic deduplication with pgvector

Eval gates: three stages, all SQL

vLLM multi-LoRA: the deploy is an HTTP POST

A/B with real traffic: measure or live deceived

Applied to typical on-premise hardware

Case 1 — RTX 4090 (24 GB) for development iteration

Case 2 — 4×H100 SXM cluster (320 GB, NVLink) for production

Position within the architecture: what this article covers and what it does not

What we have not covered (upcoming articles)

See also

References

Distribution shift between the data and `π_ref`