Continuous fine-tuning in production: from real traffic to the deployed adapter

Contents

TL;DR

Continuous fine-tuning is not “training the model every so often”. It is a closed loop where real production traffic generates the datasets, a short pipeline trains a LoRA adapter, a battery of evaluations decides whether it is promoted, and vLLM loads it without restarting. The state of the art in May 2026 has fragmented the stack: it is no longer DPO against everything, but a choice between SFT, DPO, KTO, ORPO and SimPO according to the type of signal your product captures. What has consolidated the pattern is the combination of PostgreSQL 18 + pgvector 0.8 as the nervous system of the pipeline, traffic capture, dataset versioning, eval results, adapter registry, together with vLLM multi-LoRA hot-swap, which turns deployment into an HTTP call. This article takes the cycle apart with concrete schemas, real queries, and the numbers it costs on an RTX 4090 against a 4×H100 cluster.

You are here: Tune + Retrain

This post crosses two stages of the six-stage LLMOps pipeline: the decision to train a new adapter (the Tune stage) is triggered by the signals from Observe that travel through the Retrain stage until the loop closes. The post takes apart the complete circuit between the two boxes.

Picture a neighbourhood restaurant with a signature dish that works, but the chef knows it can be tuned. Every night things happen:

Some diners leave part of the dish: a weak signal that something did not quite land.
Others ask for another version (“could you put less salt in it?”): an explicit, directional signal.
Others finish the dish and come back the following week: the only signal that really matters, but it arrives late.
And a select group offers an opinion without being asked, usually a negative one.

The chef does not redo the menu every night. He does something more interesting: he writes down in a notebook the dishes served, the returns, the changes requested, the tips. Every so often, he reads the whole notebook, decides on minimal adjustments to a recipe, tries the new version at a private table with his staff, and only if they take to it does he add it to the next day’s menu. Sometimes he even serves two different versions of the dish to different tables for a week, measures what happens, and picks one.

That is continuous fine-tuning. The notebook is Postgres. The dish is the base model. The notes are feedback signals, explicit and implicit. The “minimal adjustment” is a 30 MB LoRA adapter. The private table is the battery of automated evaluations. The next day’s menu is vLLM with multi-LoRA hot-swap, which loads the new adapter without restarting the service. Serving two versions to different tables is A/B testing with real traffic.

The analogy is exact on one critical point: the chef does not throw away the original recipe. He keeps the base recipe and holds a separate notebook with the “modifications that work well for the neighbourhood regulars”. That notebook is the LoRA adapter: on top of the base model, not in its place.

The cycle, taken apart

Before going into components, it helps to fix the complete flow. These seven steps are what any serious team replicates with variations:

The cycle lasts between 1 and 4 weeks in real production. What changes between teams is the tempo (faster in assistant chat, slower in regulated banking) and the details of each step. The structure is the same.

Why continuous fine-tuning (and why it is not RAG)

Before going deeper, a distinction that people still confuse. Fine-tuning is for form, not for facts. If your problem is that the model does not know the customer’s rates or the updated catalogue, do not fine-tune: use RAG. If your problem is that the model responds in a tone that does not fit, does not respect your JSON format, refuses legitimate cases or invents structure, then fine-tuning is the answer.

In 2026 the boundary is well established by community practice:

Observed problem	Solution
The model does not know X (X changes weekly)	RAG
The model knows X but responds badly in tone or format	SFT fine-tuning
There are two ways of responding and I prefer one over the other	Fine-tuning with preferences (DPO/KTO/ORPO/SimPO)
The model reasons badly in a verifiable domain (code, maths)	RL with verifiable reward (GRPO/DAPO)
The model is competent, it just needs a memory of facts	RAG, not fine-tuning

Continuous fine-tuning is the disciplined version of the second and third cases. The key word is continuous: it is not a one-off “we aligned the model” event, it is a process that runs every time the traffic distribution drifts far enough, or new use cases appear.

The four techniques according to the signal you capture

The most important change of the last 12 months has been the end of DPO’s monopoly. In 2024 every team doing alignment used DPO with (chosen, rejected) pairs. In 2026 the choice is finer and depends on what the signal you collect in your product looks like:

Real signal in the product	Recommended technique	Why
Labelled correct examples (input → expected output)	SFT + LoRA	Still the baseline. 500-5,000 examples are enough for style.
Explicit `(chosen, rejected)` pairs	DPO or SimPO	SimPO removes the reference model → 50 % less VRAM in training.
Loose 👍 / 👎 on responses	KTO	The method that fits real telemetry most naturally.
SFT and preferences in a single pass	ORPO	A single model in memory, avoids drift between phases.
Verifiable reward (tests, solutions)	GRPO / DAPO	Reasoning, not chat. A different world.

The practical rule: design the feedback capture in the product thinking about which method you will be able to use afterwards. If your UI only has 👍/👎, you force the path to KTO. If you add a “regenerate response” button, you unlock DPO from regenerate-as-rejected (we will see it below). If you add an “edit response” button, the edited response becomes high-quality direct SFT.

There is a cost detail that gets little publicity. DPO needs to keep two models in memory: the one you are training and the reference one. SimPO removes that second model. ORPO does too. For a Llama 3 8B in BF16 this is the difference between needing ~32 GB of active VRAM during training (DPO) or ~16 GB (SimPO/ORPO). It is the difference between the training fitting on an RTX 4090 with aggressive QLoRA, or not fitting without offload.

Postgres as the nervous system of the pipeline

Here is the strong technical opinion of this article, and it is one worth defending with data: Postgres 18 + pgvector 0.8 + an S3/MinIO bucket for the weights is enough for the whole pipeline. You do not need MLflow, you do not need lakeFS, you do not need DVC.

This is not ideological minimalism. It is about three concrete advantages that no alternative stack matches in the on-premise scenario with compliance:

A single source of truth, a single authorisation model. The ACLs you already have for Postgres cover the training data, the eval results, the adapter registry and the audit log. You do not multiply control planes.
SQL as the pipeline’s universal language. The query that generates the dataset, the eval gate predicate, the A/B traffic assignment, the promotion decision: it is all SQL. Your team already knows SQL.
Audit and cryptographic reproducibility for free. The pg_audit and pgcrypto extensions, combined with set_hash over the dataset, give you cryptographic traceability with no additional code. It is a subject that would fill an article of its own.

A concrete schema

We start with the traffic table, partitioned by week so that DROP PARTITION is cheap:

CREATE TABLE obs.inference_log (
  id            BIGSERIAL,
  request_id    UUID NOT NULL,
  tenant_id     INT NOT NULL,
  user_hash     BYTEA,                    -- GDPR pseudonymisation
  adapter_id    TEXT NOT NULL,            -- e.g. "support-es-v4.1"
  experiment    TEXT,                     -- e.g. "rerank-v2-canary"
  variant       CHAR(1),                  -- 'A' | 'B' | NULL
  messages      JSONB NOT NULL,
  completion    TEXT,
  ttft_ms       INT,
  tokens_in     INT,
  tokens_out    INT,
  -- feedback signals
  fb_explicit   SMALLINT,                 -- -1/0/+1 (KTO-ready)
  fb_regen      BOOLEAN DEFAULT false,    -- user regenerated -> DPO-rejected
  fb_edited     BOOLEAN DEFAULT false,    -- user edited -> SFT golden
  parent_id     BIGINT,                   -- self-reference for regenerate
  -- vector and meta
  embedding     HALFVEC(1024),            -- pgvector 0.8, half the RAM
  pii_flags     SMALLINT DEFAULT 0,       -- bitmask
  created_at    TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);

CREATE TABLE obs.inference_log_2026w21 PARTITION OF obs.inference_log
  FOR VALUES FROM ('2026-05-18') TO ('2026-05-25');

CREATE INDEX ON obs.inference_log_2026w21
  USING hnsw (embedding halfvec_cosine_ops);
CREATE INDEX ON obs.inference_log_2026w21
  (tenant_id, adapter_id, created_at);

Three decisions deserve a note:

HALFVEC(1024). Native FP16 vectors in pgvector 0.8. Half the RAM and disk, with a loss of precision that is irrelevant for semantic deduplication. This alone, at a scale of millions of rows, saves between 4 and 8 GB.

Weekly partitioning by time range. At 90 days, DROP TABLE obs.inference_log_2026wXX frees space in milliseconds without a prolonged lock. Autovacuum never touches frozen partitions again.

Self-referencing parent_id. The user regenerates the response → a new row is inserted with parent_id pointing at the previous one. That will give us a DPO dataset without touching the UX.

The adapter registry

CREATE TABLE serve.adapter (
  id                  TEXT PRIMARY KEY,
  base_model          TEXT NOT NULL,
  rank                INT, alpha INT,
  target_modules      JSONB,
  method              TEXT,              -- 'sft'|'dpo'|'kto'|'orpo'|'simpo'
  training_run_id     UUID,
  dataset_snapshot_id UUID,
  weights_uri         TEXT,              -- s3://.../v4.2.safetensors
  eval_summary        JSONB,
  status              TEXT NOT NULL,     -- 'training'|'canary'|'prod'|'retired'
  traffic_pct         NUMERIC(5,2) DEFAULT 0,
  promoted_at         TIMESTAMPTZ
);

The vLLM router reads this table with a TTL of a few seconds. An UPDATE serve.adapter SET status='prod', traffic_pct=100 WHERE id='v4.2' is a promotion. An UPDATE ... SET status='retired' is a rollback. The audit of who did what and when comes from pg_audit without writing a single additional line of code.

Generating DPO and KTO datasets from real traffic

This is where the elegance of the schema pays off. The dataset is not a static file: it is a materialised view built with SQL over obs.inference_log.

KTO dataset from 👍/👎

KTO is the method that best fits the signal any decent chat product captures. The query:

CREATE MATERIALIZED VIEW train.kto_v3_candidate AS
SELECT
  messages                                            AS prompt,
  completion                                          AS response,
  CASE WHEN fb_explicit > 0 THEN true ELSE false END AS label,
  adapter_id, created_at
FROM obs.inference_log
WHERE fb_explicit != 0
  AND created_at > now() - interval '60 days'
  AND pii_flags = 0
  AND tenant_id IN (SELECT id FROM tenant WHERE consent_training);

Simple. Every row with explicit feedback becomes an example (prompt, response, desirable_yes_no). KTO trains directly on this signal, with no need to build pairs.

DPO dataset from “regenerate”

The trick that is worth this article on its own. When the user presses “regenerate response”, they are giving an extraordinarily strong signal: the first response was no good to them. If the second is neither regenerated nor rated negatively, we assume it was. That is a DPO pair without a single extra click in the UI:

CREATE MATERIALIZED VIEW train.dpo_v3_candidate AS
SELECT
  rej.messages                                AS prompt,
  cho.completion                              AS chosen,
  rej.completion                              AS rejected,
  rej.adapter_id
FROM obs.inference_log rej
JOIN obs.inference_log cho ON cho.parent_id = rej.id
WHERE rej.fb_regen = true
  AND cho.fb_explicit >= 0
  -- length bias mitigation in classic DPO
  AND cho.tokens_out BETWEEN rej.tokens_out * 0.5
                          AND rej.tokens_out * 2.5;

The clause on lengths is the cheap cure for the length bias documented in DPO. Without it, the model learns that “longer = better” because the responses the user accepts tend to be slightly longer. With SimPO or ORPO this filter is optional; with classic DPO it is necessary.

Semantic deduplication with pgvector

Before training, dedup. Two almost identical prompts in the dataset are noise that biases the model:

WITH ranked AS (
  SELECT id, embedding,
         row_number() OVER (
           PARTITION BY hashtext(messages::text)
           ORDER BY fb_explicit DESC, created_at DESC
         ) AS rn
  FROM obs.inference_log
  WHERE created_at > now() - interval '60 days'
)
DELETE FROM train.kto_v3_candidate kto
USING ranked r
WHERE r.rn > 1 AND kto.id = r.id;

And for the semantic duplicates (paraphrases) we use pgvector 0.8 directly with an iterative index scan:

-- Find near-duplicates of any given example
SELECT id, messages, embedding <=> $1 AS dist
FROM obs.inference_log
WHERE created_at > now() - interval '60 days'
  AND embedding <=> $1 < 0.05
ORDER BY embedding <=> $1
LIMIT 50;

The iterative scan is a key improvement in pgvector 0.8: before, the HNSW index could return fewer results than requested when there were additional filters (WHERE); now it iterates until it meets the limit. Without that improvement, curation queries over datasets of millions of rows were unworkable without a brutal pre-filter.

Eval gates: three stages, all SQL

The most common mistake when implementing continuous fine-tuning is skipping or watering down the eval gates. That turns the cycle into a roulette wheel. The pattern that works in 2026 is three stages, each with a different latency/coverage trade-off:

And this is where Postgres shines again: the promotion gate is expressed as an SQL predicate. Nothing more:

CREATE TABLE train.eval_result (
  adapter_id  TEXT REFERENCES serve.adapter(id),
  suite_id    TEXT,                    -- 'safety-es', 'support-helpfulness'
  metric      TEXT,
  score       NUMERIC,
  judge_model TEXT,
  judged_at   TIMESTAMPTZ DEFAULT now(),
  PRIMARY KEY (adapter_id, suite_id, metric)
);

CREATE OR REPLACE FUNCTION serve.can_promote(candidate TEXT, current TEXT)
RETURNS BOOLEAN AS $$
  SELECT NOT EXISTS (
    SELECT 1
    FROM train.eval_result c
    JOIN train.eval_result p USING (suite_id, metric)
    WHERE c.adapter_id = candidate
      AND p.adapter_id = current
      AND suite_id IN ('safety-es','support-helpfulness','refusal-rate')
      AND c.score < p.score * 0.98     -- 2 % tolerance
  );
$$ LANGUAGE sql STABLE;

An SQL function as the gate. Usable from CI with psql -c "SELECT serve.can_promote('v4.2','v4.1')" and an exit code of 0/1. No orchestrator needed, no specific UI needed. The audit stays in the Postgres log.

vLLM multi-LoRA: the deploy is an HTTP POST

Two years ago, deploying a new fine-tune meant rotating inference pods. Today it is an HTTP call. vLLM 0.7+ supports loading and unloading LoRA adapters hot, keeping several resident in VRAM and choosing the right one per request.

Server configuration:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --max-loras 4 \
  --max-lora-rank 64 \
  --env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Deploying a new adapter:

curl -X POST http://localhost:8000/v1/load_lora_adapter \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "support-es-v4.2",
    "lora_path": "/mnt/adapters/support-es-v4.2"
  }'

From that moment on, requests including "model": "support-es-v4.2" are served with that adapter applied on top of the base model. Switching between adapters has negligible latency (the most recent research on Activated LoRA takes this to levels where the switching cost is invisible).

This changes operations substantially. Deploying a new fine-tune stops being an infrastructure event and becomes a state change in Postgres. The router queries the serve.adapter table, sees that v4.2 is in canary with traffic_pct=5, and directs 5 % of requests to the new adapter. The exact route of that 5 % is decided with deterministic hashing of the user_id so that the same user always sees the same variant (sticky):

-- No assignment table, no additional state
-- the variant is computed in-place in SQL or in the router:
SELECT
  CASE WHEN hashtext($user_id || $experiment) % 100
       < (SELECT traffic_pct FROM serve.adapter WHERE id = $candidate)
       THEN $candidate ELSE $current END AS adapter_id;

A/B with real traffic: measure or live deceived

The eval gates measure against fixed benchmarks. That is necessary but insufficient. Reality is only measured with real traffic. Once the adapter is in canary, what matters are the online metrics measured over obs.inference_log for each variant:

SELECT
  adapter_id,
  COUNT(*)                                  AS n,
  AVG(fb_explicit)                          AS mean_score,
  STDDEV(fb_explicit) / SQRT(COUNT(*))      AS sem,
  AVG(ttft_ms)                              AS ttft_avg,
  percentile_cont(0.5) WITHIN GROUP
    (ORDER BY ttft_ms)                      AS ttft_p50,
  percentile_cont(0.95) WITHIN GROUP
    (ORDER BY ttft_ms)                      AS ttft_p95,
  AVG(CASE WHEN fb_regen THEN 1 ELSE 0 END) AS regen_rate
FROM obs.inference_log
WHERE experiment = $1
  AND created_at > now() - interval '7 days'
GROUP BY adapter_id;

What to look at: explicit feedback, latency (TTFT, p50, p95), regeneration rate. An adapter that raises the mean feedback but also raises the regeneration rate is suspicious, probably it is responding in a flashier but less useful way. An adapter that lowers latency but lowers feedback may be worth studying: it may be being more concise than it should.

Promotion to prod happens when, after 24-72 hours in canary, the candidate adapter beats the current one on at least one key metric without degrading the others. Again: it is an UPDATE in Postgres.

Applied to typical on-premise hardware

Let us come down to two representative configurations, one for iteration and one for production.

Case 1 — RTX 4090 (24 GB) for development iteration

An RTX 4090 with 4-bit QLoRA can train adapters on an 8B model without trouble. The VRAM budget combines four components; the KV cache during the intermediate evaluations is not negligible and deserves an explicit margin:

8B base model in 4-bit:      ~5 GB
Activations + gradients:     ~8 GB (depends on batch and context)
Optimizer state (LoRA r=16): ~0.5 GB
KV cache during eval:        ~2 GB
Safety margin:               ~8 GB

Typical times (an estimate based on community benchmarks; measure in the lab before promising anything):

Dataset	Technique	Adapter rank	Approximate time
1,000 SFT examples	LoRA r=16	16	20-40 min
5,000 SFT examples	LoRA r=32	32	2-4 h
2,000 DPO pairs	LoRA r=16	16	1-2 h
5,000 KTO examples	LoRA r=32	32	3-5 h

This puts the iteration cycle, a change in the dataset, retrain, eval, look at the number, in the range of a working day. Enough to validate a hypothesis before moving anything to the production cluster.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink) for production

With a cluster of this order the whole picture changes. You can:

Train LoRA on a 70B in BF16 with no quantisation with tensor parallel = 4.
Do full DPO with a resident reference model when the reference is quantised to FP8, or move to SimPO / ORPO, which remove that intermediate model and simplify VRAM planning (see the table of techniques above).
Support multi-tenant fine-tuning: several customer adapters training in parallel in separate logical pipelines, each isolated in a different Postgres partition with its own ACLs.
Serve multi-LoRA with --max-loras 8 on top of the base model without concurrency lowering throughput perceptibly.

The practical budgeting rule: over a 12-month horizon, a team with this cluster can run ~150-200 continuous fine-tuning cycles (training + eval + canary + promotion or discard) if the discipline around the dataset and the eval gates is strict. If it is not, it will run twice as many but with half the usefulness.

Position within the architecture: what this article covers and what it does not

To place the scope: the cycle drawn at the start has seven boxes, all of them covered here in their mechanics. Three cross-cutting layers are deliberately left out, and they are the ones that end up separating a pipeline that works technically from one that survives an audit:

Cryptographic provenance and traceability. We have mentioned dataset_snapshot and pg_audit, but the complete mechanics, the set_hash over the examples, the integration with the EU AI Act, the frozen query_sql as proof of what trained the model, is enough for a whole analysis.
Judge calibration. We have assumed LLM-as-judge works. It needs calibrating against a human rubric on at least 100 cases per critical suite before trusting it. Without that calibration, the eval gates are theatre.
The forgetting problem. What happens if a user exercises their GDPR right to be forgotten and their interactions were part of the dataset of an adapter already in production? There is no clean solution. There are options, incremental retrain, sample-level machine unlearning, negative LoRA, and it is worth knowing them before a customer asks.

What we have not covered (upcoming articles)

Cryptographic provenance on Postgres: how set_hash and a frozen query_sql compose an auditable chain of custody under the EU AI Act.
Honest judge calibration: why score > 0.85 means nothing without a human baseline, and how to build that baseline without it costing a month of work.
The forgetting problem in adapters: sample-level machine unlearning, incremental retrain and other techniques for answering GDPR without throwing the adapter away.
Online DPO and on-policy continual learning: the state of 2026 research (Fast-Slow Chasing, RLOO, iterative on-policy) and why it is not production yet.

References

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022).
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023).
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
Hong et al., ORPO: Monolithic Preference Optimization without Reference Model (2024).
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM.
Official vLLM Multi-LoRA documentation: https://docs.vllm.ai/en/stable/features/lora/.
Official pgvector 0.8 documentation: https://github.com/pgvector/pgvector.
TRL (HuggingFace) docs: https://huggingface.co/docs/trl.
EU AI Act, consolidated text and application timeline: https://artificialintelligenceact.eu/.