Fine-Tuning on lo0 — Tech Blog

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

Tue, 09 Jun 2026 03:00:00 +0000

This is the operational companion to QLoRA and multi-LoRA at the limit on small models. That post takes apart the why, NF4, double quantisation, paged optimizers, the adapter maths; this is the how, with commands you copy and paste. If you have not read the fundamentals one, read it first: here we assume you know what an adapter is, why the base lives in 4-bit and why the gradient only touches the adapter.

TL;DR

A reproducible procedure in five phases: (1) pin the environment with fixed versions; (2) prepare the dataset in chat format; (3) train the QLoRA adapter with TRL + PEFT on an RTX 4090 (24 GB, Ada Lovelace) using gradient checkpointing, gradient accumulation and paged_adamw_8bit; (4) validate and version the adapter as an artefact of megabytes; (5) serve it in vLLM with --enable-lora, loading it hot without restarting the server and resolving it from object storage. All on-premise, on consumer hardware, without a single data item leaving the perimeter. What follows are the exact commands and the memory budget that separates “it fits” from “OOM”.

The end-to-end flow

Phase 0 — Environment and versions

QLoRA is sensitive to the versions of bitsandbytes, transformers, peft and trl: misaligned combinations give dequant errors or adapters that do not load in vLLM. Pin the environment and do not touch it mid-campaign. Reference versions as of June 2026 (check the specific ones in your index; the exact pin matters less than the coherence between them):

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Training (producer)
pip install "torch>=2.4" \
 "transformers>=4.50" \
 "peft>=0.14" \
 "trl>=0.15" \
 "bitsandbytes>=0.45" \
 "accelerate>=1.2" \
 "datasets>=3.2"

# Serving (consumer) — in its own environment/image
pip install "vllm>=0.8"

What each piece does and why it is pinned:

Package	Role in the flow	Why the version matters
`torch`	tensor runtime and CUDA kernels	the CUDA ABI has to match the driver and `bitsandbytes`; a major jump breaks the 4-bit kernels.
`transformers`	loads the base, the tokenizer and the `chat_template`	it has to know the architecture of the SLM you use; a new model needs a version that supports it.
`peft`	implements LoRA/QLoRA: injects the `A,B` matrices and writes the `adapter_config.json`	that `adapter_config.json` is what vLLM reads when serving; old versions write fields the serving side does not understand.
`trl`	the `SFTTrainer`: the supervised training loop	it integrates `peft` natively; its API (`SFTConfig`) changes between versions, hence the pin.
`bitsandbytes`	the NF4 quantisation and `paged_adamw_8bit`	the most sensitive piece: a badly compiled binary gives corrupt dequant or hangs on the first step.
`accelerate`	orchestrates device, mixed precision and `device_map`	the silent backend of almost everything; misaligning it with `transformers` gives cryptic errors.
`datasets`	loads the JSONL (and allows streaming if the corpus is large)	not very sensitive; any recent 3.x will do.
`vllm`	the multi-LoRA serving	separate environment or image: do not mix its stack with the training `bitsandbytes`.

The golden rule: coherence between the four above (transformers, peft, trl, bitsandbytes) matters more than the exact number of each. Pin them when starting a campaign and do not move them until it closes.

Check that the GPU and CUDA are healthy before starting; a badly compiled bitsandbytes shows up late:

python -c "import torch, bitsandbytes; print(torch.cuda.get_device_name(0), torch.cuda.is_available())"
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

For 100 % sovereignty: download the base once from your internal Hugging Face mirror (or a MinIO with the weights) and export HF_HOME to a local volume. Nothing in this flow needs to leave the perimeter.

Phase 1 — Preparing the dataset

The canonical format for a conversational task is JSONL, one conversation per line, with the model’s chat template. Do not invent your own format: use the base tokenizer’s chat_template, because any mismatch between how you train and how you serve degrades quality silently.

{"messages":[{"role":"system","content":"You are a network support assistant."},{"role":"user","content":"The north wing AP does not come up after the outage."},{"role":"assistant","content":"First confirm the PoE on the port..."}]}
{"messages":[{"role":"user","content":"Generate the VLAN change for customer 42."},{"role":"assistant","content":"interface GigabitEthernet0/3\n switchport access vlan 42..."}]}

What each field is and why:

Field	What it is	Operational note
`messages`	the complete conversation, a list of turns	one conversation per JSONL line; it is what `apply_chat_template` turns into tokens.
`role`	who is speaking: `system`, `user`, `assistant`	the adapter learns to produce the `assistant` turns; the `user`/`system` ones are context, not target.
`content`	the text of the turn	the `system` one sets the persona/task; keep it identical to the one you will use in production or the adapter drifts out of alignment.

Operational rules that save grief: watch the ratio of examples (a well-curated narrow-task dataset of 2,000-20,000 examples performs better than 200,000 noisy ones), deduplicate, and set aside 5-10 % as a validation split that does NOT go into training. Building the corpus from production signal is covered by Retrain: closing the loop.

Phase 2 — The training script

A minimal, complete script with TRL + PEFT. It trains an r=8 adapter on an 8B SLM quantised to NF4. Each block has its rationale commented.

# train_qlora.py
import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen3-8B" # or whichever SLM you serve; ALWAYS use the same one in train and serve
OUT = "adapters/soporte-redes-v1"

# 1) Base frozen and quantised to 4-bit NF4 with double quantisation
bnb = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_quant_type="nf4", # NormalFloat, quantile-optimal for Gaussian weights
 bnb_4bit_use_double_quant=True, # quantises the scale constants -> ~0.37 bits/param less
 bnb_4bit_compute_dtype=torch.bfloat16 # the matmuls run in BF16 after dequant on the fly
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
 BASE, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map={"": 0}
)

# 2) The adapter: low rank, attention projections only (aggressive). Raise target_modules if the eval asks for it.
peft_cfg = LoraConfig(
 r=8, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

ds = load_dataset("json", data_files={"train": "data/train.jsonl",
 "eval": "data/eval.jsonl"})

# 3) Training config designed to fit in 24 GB
cfg = SFTConfig(
 output_dir=OUT,
 per_device_train_batch_size=1, # small real batch
 gradient_accumulation_steps=16, # EFFECTIVE batch = 1*16 = 16, without paying its VRAM all at once
 gradient_checkpointing=True, # recomputes activations in backward: trades compute for memory
 optim="paged_adamw_8bit", # paged optimizer: the airbag against VRAM peaks
 learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
 num_train_epochs=3, bf16=True,
 max_length=2048, # bounds the sequence: activations scale with it
 logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=200,
 report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
 train_dataset=ds["train"], eval_dataset=ds["eval"],
 processing_class=tok)
trainer.train()
trainer.save_model(OUT) # saves ONLY the adapter (MB), not the base

`BitsAndBytesConfig` — how the base is quantised

Option	What it does	Why this value / when to change it
`load_in_4bit=True`	loads the base weights in 4-bit	it is the foundation of QLoRA: without it the 8B does not even fit to train.
`bnb_4bit_quant_type="nf4"`	uses the NF4 format (quantile-optimal for Gaussian weights)	`"fp4"` exists, but NF4 performs better on transformer weights; leave NF4.
`bnb_4bit_use_double_quant=True`	quantises the scale constants themselves	saves ~0.37 bits/param (hundreds of MB on an 8B); the margin separating “it fits” from “OOM”. Leave it at `True`.
`bnb_4bit_compute_dtype=torch.bfloat16`	matmul precision after undoing the quantisation on the fly	BF16 on Ada/Hopper (4090, H100); use `float16` only on GPUs without BF16.

`LoraConfig` — the shape of the adapter

Option	What it does	Why this value / when to change it
`r=8`	adapter rank: its correction capacity	4-8 for a narrow task (aggressive); raise it to 16-64 only if the eval shows underfitting.
`lora_alpha=16`	scale factor of the delta (effective `α/r`)	common convention `α=2r`; it modulates how much the adapter “weighs” on the base.
`lora_dropout=0.05`	regularisation on the adapter	0.05-0.1 with small datasets (avoids overfit); 0 if the corpus is large.
`bias="none"`	does not train the bias terms	`"none"` is the standard; `"all"`/`"lora_only"` rarely help and cost params.
`task_type="CAUSAL_LM"`	objective/head type	fixed for a generative LLM.
`target_modules=[q,k,v,o]`	which matrices get an adapter	attention only = cheap and aggressive; add `gate_proj`/`up_proj`/`down_proj` (MLP) if the task demands rewriting more behaviour and the eval asks for it.

`SFTConfig` — the memory budget and the loop

Option	What it does	Why this value / when to change it
`per_device_train_batch_size=1`	microbatch per GPU	1 on 24 GB; the real batch is built by `gradient_accumulation_steps`.
`gradient_accumulation_steps=16`	accumulates 16 microbatches before updating	effective batch = 1×16 = 16 without paying its VRAM all at once; raise it if you shorten the sequence and want a bigger effective batch.
`gradient_checkpointing=True`	recomputes activations in the backward pass instead of storing them	essential on a 4090: ~20-30 % slower in exchange for much less VRAM.
`optim="paged_adamw_8bit"`	Adam optimizer in 8-bit + states pageable to RAM	less state VRAM and the airbag that avoids the OOM at the peaks.
`learning_rate=2e-4`	learning rate of the adapter	1e-4-3e-4 is the typical QLoRA range; adapters tolerate a higher LR than a full fine-tune.
`lr_scheduler_type="cosine"`	LR decay curve	`cosine` or `linear`; cosine usually gives a smooth drop at the end.
`warmup_ratio=0.03`	warms the LR over the first 3 % of steps	avoids the instability of the first steps.
`num_train_epochs=3`	complete passes over the dataset	1-3; watch the eval loss so as not to overfit.
`bf16=True`	compute and adapter precision	BF16 on Ada/Hopper; `fp16=True` if your GPU has no BF16.
`max_length=2048`	maximum sequence length	the #1 VRAM lever for activations: shorten it first if there is an OOM.
`eval_strategy`/`eval_steps`/`save_steps`	validation and checkpoint cadence	tune them to the dataset size; evaluating often costs time.

The four pieces that make it fit on a 4090 are: per_device_train_batch_size=1 + gradient_accumulation_steps (a large effective batch without its memory cost all at once), gradient_checkpointing=True (recomputing activations instead of storing them) and optim="paged_adamw_8bit" (paging states to RAM at the peaks). Remove any of the three with long sequences and you will see the OOM.

A declarative alternative with Axolotl if you prefer YAML over Python (same result):

base_model: Qwen/Qwen3-8B
load_in_4bit: true
adapter: qlora
lora_r: 8
lora_alpha: 16
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
sequence_len: 2048
micro_batch_size: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
optimizer: paged_adamw_8bit
learning_rate: 0.0002
num_epochs: 3
bf16: true
datasets:
 - path: data/train.jsonl
 type: chat_template

Phase 3 — Launch and monitor

# Simple launch on one GPU
python train_qlora.py

# In another terminal: watch VRAM. If it gets close to the ceiling, lower max_length or raise grad accumulation.
watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Approximate VRAM budget when training the 8B on the 4090, and what to touch when it gets tight:

Component	Approx. VRAM	Lever if there is an OOM
8B NF4 base (frozen)	~4.0 GB	— (fixed)
Adapter + grad + Adam states	~0.3-0.7 GB	lower `r`
Activations (batch × sequence)	~6-14 GB	lower `max_length`, `batch_size`; raise `grad_accum`
Dequant buffers / workspace	~1-2 GB	—

Table of quick OOM remedies, in order of cost: (1) lower max_length; (2) confirm gradient_checkpointing=True; (3) raise gradient_accumulation_steps and lower per_device_train_batch_size to 1; (4) use paged_adamw_8bit (already in the script); (5) as a last resort lower r. If after all that it still does not fit, the sequence or the model is too large for 24 GB: either you bound it, or you move up the hardware.

Phase 4 — Validating the adapter

Never promote an adapter on the strength of the training loss. Measure against the validation split you set aside and against a handful of real prompts.

# quick_eval.py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
 bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, "adapters/soporte-redes-v1") # base + adapter

msgs = [{"role": "user", "content": "The north wing AP does not come up after the outage."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(0)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0], skip_special_tokens=True))

For a serious verdict, run the adapter through your eval suite (the layer described by LLM evals) and compare against the base without an adapter and against the previous version of the adapter. Promote only if it wins on the task metric without regressing on safety/format.

Phase 5 — Versioning the adapter as an artefact

The adapter is a pair of files of a few MB (adapter_model.safetensors + adapter_config.json). Treat it as a versioned, signed, traceable artefact, not as a loose file.

# Reproducible checksum + upload to internal object storage (MinIO/S3)
sha256sum adapters/soporte-redes-v1/adapter_model.safetensors > adapters/soporte-redes-v1/SHA256
aws --endpoint-url https://minio.interno s3 cp \
 adapters/soporte-redes-v1/ s3://adapters/soporte-redes/v1/ --recursive

A convention that works: s3://adapters/<task-or-customer>/<version>/. Immutable per version, with its SHA256. Deleting a customer is deleting a prefix of a few MB, not retraining anything. Versioning 500 adapters costs what versioning 500 heavy configuration files costs.

Phase 6 — Serving in multi-LoRA with vLLM

The consumer loads one shared base and applies the adapter delta per request. Startup with static adapters declared:

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3-8B \
 --enable-lora \
 --max-loras 8 \  # max nº of DIFFERENT adapters per batch (not the total loadable)
 --max-lora-rank 8 \  # = the maximum rank of your adapters; do not inflate it (wastes memory)
 --max-cpu-loras 64 \  # adapters cached in RAM for fast swap to VRAM
 --lora-modules soporte-redes=/srv/adapters/soporte-redes/v1

Each flag, what it controls and how to size it:

Flag / variable	What it controls	How to size it
`--enable-lora`	turns on adapter support	mandatory; without it, vLLM ignores any `model` that is an adapter.
`--max-loras 8`	nº of different adapters in a single batch	more adapters per batch makes the SGMV kernels more expensive; 8-32 is reasonable. It is not the total loadable.
`--max-lora-rank 8`	maximum rank the server reserves	set it equal to the real rank of your adapters (8 here); inflating it wastes VRAM and performance.
`--max-cpu-loras 64`	adapters cached in RAM ready to page to VRAM	≥ nº of active adapters; it is the “bench” from which fast swapping happens.
`--lora-modules name=path`	declares static adapters at startup	useful for the fixed ones; leave it out if everything goes through dynamic loading/Resolver.
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`	enables the hot load/unload endpoints	essential for `/v1/load_lora_adapter`; without it, the server is static.

--max-loras limits the different adapters per batch, not how many you can have loaded; the bulk lives on CPU (--max-cpu-loras) and is paged to VRAM on demand. Set --max-lora-rank to the real rank (8 here): inflating it wastes memory and performance. Requests pick the adapter through the model field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "soporte-redes",
 "messages": [{"role":"user","content":"The north wing AP does not come up after the outage."}]
}'
# model:"Qwen/Qwen3-8B" (no adapter) uses the bare base on the same server

Hot loading of a new adapter without restarting (thanks to VLLM_ALLOW_RUNTIME_LORA_UPDATING=True):

curl -X POST http://localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" -d '{
 "lora_name": "cliente-42",
 "lora_path": "/srv/adapters/cliente-42/v3"
}'
# and to free VRAM/CPU when a customer goes idle:
curl -X POST http://localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" -d '{"lora_name":"cliente-42"}'

For multi-tenant at scale, avoid declaring hundreds of adapters by hand: the LoRAResolver resolves and loads the adapter from local storage or S3 the first time an unknown model arrives, so the server stays lean and the adapters are pulled lazily from your MinIO. The internals of how thousands of concurrent adapters are batched (SGMV kernels, unified paging, the heterogeneous gather/scatter) are in Multi-LoRA serving; this runbook only switches them on. To squeeze the base’s decode throughput on a 4090, combine this with Optimising decode in vLLM.

Serving multi-adapter vs merging per task

Two deployment architectures, and the procedure changes:

Serving multi-LoRA (the above). A shared base + N hot adapters. It is the default sovereign pattern: minimum footprint, per-customer isolation, hot-swap. Use standard QLoRA and merge nothing.

Merging per task. If you want a single quantised-and-adapted artefact per task (no adapter at runtime), do not merge a standard QLoRA adapter into the 4-bit base: the merge reintroduces precision that NF4 does not represent and on requantising you lose part of what was learnt. For that case, train with QA-LoRA (quantization-aware), which merges cleanly onto a quantised base. It is an architecture decision, not a quality one; the conceptual detail is in the fundamentals post.

Checklist of operational gotchas

Coherent chat template between training and serving. The most common and most silent mismatch: you train with one chat_template and serve with another. Use the base’s on both sides.
Exactly the same base (revision included) in train and serve. An adapter trained on Qwen3-8B is not valid on another revision of the model.
--max-lora-rank ≥ the rank of ALL the adapters served together, but no more: inflating it wastes VRAM.
KV budget vs --max-loras. The bottleneck in serving is not the adapters (MB), it is the KV cache and concurrency; see Inverted roofline for the SLM regime.
r too low = underfitting if the task demands rewriting a lot of behaviour. Raise r only if the eval asks for it.
Do not promote on training loss. Validate against the reserved split + real prompts + a safety regression.
Version and make immutable every adapter with its SHA256; never overwrite a served version.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB) the same machine is producer and consumer: you train a customer’s adapter in hours and serve it on the same server on top of the shared base. It is the canonical case for multi-tenant demos and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8) QLoRA stops being necessary in order to fit, but it serves to parallelise production (several adapter jobs at once) and to keep the quantised format consistent between training and serious serving of hundreds of concurrent adapters. The base can run in native FP8; the mechanics of the runbook do not change, only the scale.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hugging Face TRL — PEFT integration (SFTTrainer + QLoRA): https://huggingface.co/docs/trl/peft_integration
Hugging Face PEFT: https://github.com/huggingface/peft
bitsandbytes: https://github.com/bitsandbytes-foundation/bitsandbytes
vLLM — LoRA Adapters (serving, dynamic loading, LoRAResolver): https://docs.vllm.ai/en/stable/features/lora/
Axolotl: https://github.com/axolotl-ai-cloud/axolotl
Xu, Y. et al. QA-LoRA: Quantization-Aware Low-Rank Adaptation. ICLR 2024. https://arxiv.org/abs/2309.14717

QLoRA and multi-LoRA at the limit on small models

Tue, 09 Jun 2026 02:30:00 +0000

This post is the training-side companion to Multi-LoRA serving. That one takes apart the consumer, how hundreds of concurrent adapters are served with SGMV kernels and unified paging; this one takes apart the producer, how an adapter is trained on top of a quantised base on a single GPU, and why the pattern “one frozen base SLM + N low-rank adapters” is the natural fit for small models. We do not repeat the serving internals here; we assume you have read them.

TL;DR

QLoRA (Dettmers et al., NeurIPS 2023) solves a specific problem: fine-tuning a model without having the VRAM to hold its weights in BF16, its gradients and the optimiser states. The idea has three pieces. One: freeze the base and quantise it to 4-bit with a new format, NF4 (NormalFloat 4-bit), quantile-optimal for weights distributed almost like a Gaussian. Two: do not train the base, not a single one of its weights moves, but rather a pair of small LoRA matrices in BF16 plugged in parallel; the gradient flows only through that adapter. Three: two memory tricks, double quantisation (quantising the quantisation constants themselves) and paged optimizers (optimiser states that page out to RAM when VRAM gets tight). The measurable operational result: a 3-8B SLM is fine-tuned on an RTX 4090 (24 GB, Ada Lovelace), not on a cluster. And because the output of training is an adapter measured in megabytes, not gigabytes, the pattern that emerges is a single frozen base SLM in 4-bit plus N adapters, one per customer, domain or task, served on top of the shared base with the stack we already covered in multi-LoRA serving. Per-customer isolation, minimum footprint, sovereign deployment.

The analogy: the frozen guitar and the swappable pedalboard

Think of a session guitarist recording for very different clients: a jazz record, a corporate jingle, a metal track. He has one single guitar, his trusted instrument, set up, with a base tone he knows by heart. What he does not do is buy a new guitar for every song. What he does is keep a pedalboard: a distortion pedal, a chorus, a delay. For each track he plugs in the pedal that fits, and the same guitar sounds completely different.

The mapping is exact:

The guitar = the base SLM. A single copy, set up at the factory, frozen. In QLoRA it is also kept in a compressed case: quantised to 4-bit. You do not touch it: you do not change its pickups or adjust the neck. It weighs what it weighs and there it stays.
Each pedal = a LoRA adapter. Small, cheap, specific to one sound. You train it for a task and put it in a drawer.
Training QLoRA = designing a new pedal by listening to the (frozen) guitar through it, adjusting only the pedal’s pots until it sounds the way you want. The guitar’s base tone is not modified; you learn the correction the pedal applies on top.
Serving multi-LoRA (Multi-LoRA serving) = having the whole pedalboard set up on stage and choosing the right pedal per note, per request. The guitar is the same; what changes between requests is which pedal is active.

The analogy holds down to the detail that confuses people most: the training gradient only “touches” the pedal. The guitar is frozen in its compressed case; learning does not move it. That is what lets the base live in 4-bit throughout the fine-tuning without the quantisation getting in the way: no gradient is ever computed for it.

The bare mechanism: LoRA, and why you can train on a 4-bit base

A minimal reminder of LoRA (Hu et al., ICLR 2022). An adapter modifies a matrix W of the base by adding a low-rank product to it:

$$W' = W + B A, \qquad A \in \mathbb{R}^{r \times d}, \quad B \in \mathbb{R}^{d \times r}$$

with r the rank, much smaller than d. In the forward pass BA is not materialised; what is computed is:

$$y = W x + B(A x)$$

The base compute (Wx) happens just the same; the adapter adds two cheap matmuls. The key to QLoRA is who receives a gradient. The base W is frozen: ∂L/∂W is neither computed nor stored. Only A and B are trainable. That is why W can live quantised to 4-bit without trouble: in the forward pass the quantisation is undone on the fly to compute Wx (dequant → matmul in BF16), but since W is never updated it does not need the precision of a trainable weight. The adapter A, B is in BF16, and it is the only path the gradient flows through.

This is what breaks the memory wall. In a full fine-tuning you need, per weight: the weight (2 bytes BF16), its gradient (2 bytes), and the two Adam states (momentum and variance, typically 4+4 bytes in FP32), on the order of 12-16 bytes per trainable parameter. With QLoRA, the base weights take 0.5 bytes (4-bit) and have no gradient and no optimiser states. Only the few million adapter parameters pay the 16-byte cost. For an 8B, that is the difference between ~130 GB and fitting in 24 GB.

NF4: why a new format instead of INT4

QLoRA does not use linear INT4 for the base, but NF4 (NormalFloat 4-bit). The intuition: the weights of a trained transformer are distributed, empirically, very close to a zero-centred Gaussian. INT4 spreads its 16 levels uniformly across the range, which wastes levels in the tails (where there are almost no weights) and leaves few in the centre (where they pile up). NF4 spreads the 16 levels according to the quantiles of a normal distribution: more levels where there is more probability mass. It is, by construction, information-theoretically optimal for exactly Gaussian data, each level covers roughly the same number of weights. It is also symmetric about zero and guarantees an exact representation of 0 (important for sparsity and padding). The detail of the quantisation formats is in Quantisation for LLM inference; here the idea that NF4 spends its bits where the weights are is enough.

Double quantisation and paged optimizers

Quantising to 4-bit is not entirely free: for each block of weights (typically 64) you need to store a scale constant in FP32 so the quantisation can be undone. Those constants weigh something. With blocks of 64 and one FP32 scale (32 bits) per block, that is 32/64 = 0.5 bits per parameter in metadata alone, a 12.5 % overhead on top of the 4 useful bits. Double quantisation attacks that: it quantises the scale constants themselves (to 8-bit, in blocks of 256), bringing the overhead down to ~0.127 bits/param. Quantising the quantisation sounds recursive and it is; the saving is small in absolute terms (~0.37 bits/param) but on an 8B that is hundreds of MB, which is exactly the margin separating “it fits” from “it does not fit” on a 4090.

Paged optimizers attack the memory peaks. During training, certain moments, a batch with a very long sequence, a large activation, push VRAM close to the limit and blow up with an OOM. The idea, borrowed from operating system paging, is to allocate the optimiser states in NVIDIA unified memory: when VRAM gets tight, those pages are evicted to host RAM automatically and brought back when needed. It speeds nothing up; it avoids the crash at the peaks. It turns an “intermittent OOM” into “a bit slower at the worst moments”, which for an unattended training run on a single GPU is the difference between finishing and not finishing.

“Aggressive” training: very low rank and QA-LoRA

“Aggressive” in this context means two things, sometimes combined.

Very low rank (r = 4-8). The rank is the bottleneck of the correction: how much “capacity” the adapter has to steer the base away from itself. A high rank (64, 128) brings the adapter closer to a full fine-tuning but weighs more and takes longer to train. For an SLM adapted to a narrow, well-defined task, an output format, a lexical domain, a response style, a rank of 4-8 is usually enough, and the resulting adapter weighs a fraction. The risk of a low rank is underfitting: if the task demands rewriting a lot of the base’s behaviour, r=4 falls short. The honest rule is empirical: raise the rank only if the eval asks for it, not “just in case”. On small SLMs, where the base has less spare capacity, a low rank tends to work proportionally better than on large models, but this depends on the task and has to be measured, not assumed.

QA-LoRA (quantization-aware LoRA, Xu et al., arXiv:2309.14717). There is a subtle friction in standard QLoRA: you train the adapter in BF16 against a 4-bit base, but if you then want to merge the adapter into the base (W' = W + BA) to serve a clean quantised model, the merge reintroduces precision that the 4-bit format cannot represent, and on requantising you lose part of what was learnt. QA-LoRA trains the adapter aware of the target’s quantisation: it balances the degrees of freedom of the quantisation and of the adaptation (with group-wise quantisation) so that, when it finishes, the adapter merges cleanly into a quantised base without a requantisation step that degrades it. The result is a final model that is quantised-plus-adapted, with no separate adapter at runtime, useful when you want a single deployable artefact per task instead of the shared-base + adapters pattern. The choice between “QLoRA + serve multi-adapter” and “QA-LoRA + merge per task” is a deployment architecture decision, not one of pure quality.

The maths that matters

Three calculations drive any QLoRA decision on SLMs.

Adapter parameters. For each target matrix of dimension d with rank r, the adapter contributes A (r×d) plus B (d×r), that is 2·r·d parameters. Summing over the target matrices and multiplying by the number of layers:

$$\text{params}_{\text{adapter}} = L \cdot \sum_{\text{matrices}} 2 \cdot r \cdot d$$

Worked example — Llama-3-8B, attention (q, k, v, o), d = 4096, L = 32 layers, r = 8. Taking the four attention projections with the same d = 4096 (a simplification; in Llama-3 K and V are narrower because of GQA, which gives even fewer params):

$$\text{params} \approx 32 \cdot 4 \cdot (2 \cdot 8 \cdot 4096) = 32 \cdot 4 \cdot 65\,536 \approx 8.4\text{M params}$$

In BF16 (2 bytes/param): 8.4M · 2 ≈ 16.8 MB ≈ ~17 MB. Seventeen megabytes. Compare that with the base: an 8B in NF4 takes 8\text{G} · 0.5\,\text{bytes} ≈ 4\text{ GB} (plus the small constants overhead after double quantisation). The adapter is 0.4 % of the size of the quantised base. This is what makes having hundreds operationally trivial: an adapter is not a model, it is almost a heavy configuration file.

How many adapters fit on a 4090 after the base + KV? Budget for an RTX 4090 (24 GB): 8B NF4 base ~4 GB, leave ~5 GB for KV cache and inference activations at moderate concurrency → that leaves ~15 GB free (being conservative, call it ~12-15 GB). With ~17 MB adapters (r=8, attention-only):

$$\frac{15\,000\ \text{MB}}{17\ \text{MB/adapter}} \approx 880 \text{ adapters}$$

On the order of thousands if you cut the reserved KV cache or use rank 4 (~8.5 MB/adapter → ~1750 in 15 GB). The bottleneck is never the adapters’ space; it is the KV cache and concurrency. For the details of how those thousands are served concurrently, the heterogeneous batching, the unified paging, the SGMV kernels, see Multi-LoRA serving. The relevant summary here: the adapter compute is nearly free (low rank, two thin matmuls); the serving performance challenge is not that compute but the gather/scatter of the right adapters per batch row when a single batch mixes requests from different adapters. That is the consumer’s problem, not the producer’s.

QLoRA training VRAM in 24 GB. The rough budget to fine-tune the 8B on a 4090:

Component	Approx. VRAM
8B base in NF4 (frozen weights)	~4.0 GB
Adapter (BF16 params + gradient + Adam states, ~16 B/param over ~8-40M params)	~0.3-0.7 GB
Activations (depends on batch and sequence length; the variable bulk)	~6-14 GB
Dequant buffers, scales, workspace	~1-2 GB
Total	fits in 24 GB with margin

The large, variable piece is the activations, which scale with batch × sequence length. That is why real QLoRA on a 4090 is done with a small batch + gradient accumulation (simulating a large batch by accumulating gradients over microbatches) + gradient checkpointing (recomputing activations in the backward pass instead of storing them, trading compute for memory) + bounded sequences. The paged optimizers are the airbag for the activation peaks that would otherwise blow up. The claim “QLoRA fine-tunes an 8B on a 4090” is true with that configuration; with long sequences, a large batch or a high rank, it does not fit. As with any number, the methodology matters more than the headline.

The fit with small models and sovereignty

This is where QLoRA + SLM stops being a VRAM trick and becomes an architectural pattern.

An SLM (3-8B) already fits comfortably on a single GPU for inference. If on top of that the base lives in 4-bit (~4 GB for an 8B), you have memory to spare. What QLoRA enables is for that same machine, the 4090, to be both the producer and the consumer: you train a new customer’s adapter in hours, on the same class of hardware where you then serve it. The artefact that travels between “train” and “deploy” is an adapter of MB, not GB: it is versioned, signed, moved across the network, stored in MinIO/S3 without thinking about the cost.

The sovereign pattern falls out on its own:

Per-customer isolation. Each customer has their adapter, trained only on their data. The base is generic and shared; what is specific to the customer lives isolated in their (A, B) pair. Deleting a customer is deleting a file of a few MB, not retraining anything.
Minimum footprint. One base + N adapters fits where N bases would not come close. The economics of “one model per customer” (tens of GB each) are prohibitive; those of “one base + adapters” (MB each) are trivial. It is exactly the difference between the pedalboard and buying a guitar per song.
Sovereign deployment. Everything fits on-premise, on your hardware, without a single data item leaving the perimeter. Training (QLoRA on the 4090) and serving (multi-LoRA on the same base) live inside. There is no dependency on an external API for fine-tuning or for serving.

The choice between adapting by domain (one adapter per knowledge area) and retrieving by context (RAG that injects the knowledge into the prompt) is real and not mutually exclusive: the adapter changes the model’s behaviour and style, RAG changes the facts it has access to. The sibling post on aggressive RAG on small models in this series works through it; the short rule is: adapt what is stable and behavioural, retrieve what is volatile and factual.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

This is QLoRA’s natural workbench. The canonical case: a 3-8B base SLM in NF4, fine-tuning an r=8 attention-only adapter, with gradient checkpointing + gradient accumulation + paged optimizer. It trains in hours for narrow-task datasets (thousands to tens of thousands of examples), and the same machine then serves the base + tens or hundreds of adapters for multi-tenant demos and platform prototypes. The 4090 is where QLoRA went from “a paper technique” to “anyone with a consumer GPU can do it”, and that is exactly its value. The honest rule: it fits with the memory configuration described; with long sequences, a large batch or a high rank, move up the hardware.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here QLoRA stops being strictly necessary in order to fit, an 8B in BF16 has room to spare, but it remains useful for another reason: parallelising adapter production. With 320 GB and native FP8 you can train several adapters at once (one job per customer, several in parallel), or fine-tune somewhat larger models with QLoRA without TP. The consumer on this cluster is the serious setup from Multi-LoRA serving: an FP8 base + hundreds of concurrent adapters. The rule of thumb: on the 4090, QLoRA is the tool to be able to fine-tune; on the H100 cluster, it is the tool to fine-tune many at once, cheaply, keeping the quantised format consistent between training and serving.

What we have not covered

The internals of heterogeneous serving (SGMV kernels, MBGMM/MBGMV, unified paging, cold start, eviction): they are covered in full in Multi-LoRA serving. This post is deliberately the producer’s side.
DoRA and variants (magnitude-direction decomposition): they close part of the gap with full fine-tuning; different training pattern, identical serving pattern.
Sub-4-bit and ternary quantisation of the base: what happens when the base drops from NF4 to 2-bit or ternary underneath the adapter; the sibling post in the series works through it.
Collecting the fine-tuning dataset: how each adapter’s corpus is built from production feedback is in Retrain: closing the loop.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. ICLR 2024. https://arxiv.org/abs/2309.14717
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Official QLoRA / bitsandbytes repo: https://github.com/artidoro/qlora
Hugging Face PEFT (LoRA, QLoRA): https://github.com/huggingface/peft

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

Sat, 30 May 2026 14:00:00 +0200

This post complements Continuous fine-tuning in production. Continuous fine-tuning is the producer of the adapters; multi-LoRA serving is the consumer that puts them to work. Without this layer, the whole feedback cycle breaks in the last mile. It also crosses over with Modern alignment (DPO/KTO/ORPO/SimPO) (each alignment policy can live as a separate adapter) and Quantisation (a quantised base frees memory for many more adapters).

You are here: DEPLOY

TL;DR

The dominant pattern in 2026 is not one model per customer but a single general-purpose base model plus N fine LoRA adapters per task, customer, language or domain. The reason is obvious: a rank-16 LoRA over Llama-3-70B takes about 400 MB; a full fine-tune takes about 140 GB. Dozens or hundreds of adapters per base is manageable; dozens or hundreds of bases is prohibitive. What is not obvious is how to serve them concurrently without reloading weights every time the adapter changes (killing batching) or replicating the base (killing memory). The answer crystallised in 2024 with two complementary papers: S-LoRA (Sheng et al., Stanford + UC Berkeley, MLSys 2024) introduced unified paging, where adapter weights live in the same memory pool as the KV cache, both pageable, and heterogeneous batching, where a batch can hold requests with different adapters and different ranks without padding; Punica (Chen et al., UW + Duke, MLSys 2024) introduced the CUDA kernel that has become the de facto standard: SGMV (Segmented Gather Matrix-Vector multiplication), which computes Y += Σ_i X_i · A_i · B_i in a single pass, grouping requests by adapter. SGMV sits today underneath vLLM, LoRAX (Predibase), SGLang and TGI. The measurable operational result: up to 2,000 concurrent adapters on a single GPU (S-LoRA paper), up to 4× throughput vs naive vLLM and up to 30× vs HuggingFace PEFT. The price: typical overhead of 10-30 % latency per layer with an active adapter in a heterogeneous batch, practically zero when every request in the batch uses the same adapter, and 20-40 % in the worst case. This post takes apart the mechanism, the maths (memory per adapter, overhead per rank), the comparison table of implementations, the pitfalls (cold start, mismatched rank, fragmentation) and the real economics on an H100 with a Llama-3-70B FP8 base + 200 adapters.

The analogy: the single translator with a thousand glossaries

Imagine a specialised translation agency with a single senior translator, brilliant, fluent in fifteen languages and across all the general technical domains. That translator is expensive to hire and expensive to train: it took years of formation and experience that is not easily replicated. But the agency receives texts from very different clients: a law firm using legal terminology specific to its jurisdiction, a manufacturer with an internal parts nomenclature, a hospital with its own clinical abbreviations. Every client has its own jargon.

The agency does not hire one translator per client; that would be ridiculous, 90 % of the work is shared. What it does is keep one glossary per client: a small notebook, easy to update, containing the specific terms and how they are translated for that client. When the translator receives a text, they open the glossary for the client in question and work with it alongside. As they translate each word, they first check whether it is in the glossary; if it is, they use the specific version; if not, they use their general knowledge.

The glossaries live on a shared shelf, ordered by recent use: the most consulted ones within reach, the old ones in the archive. When a new client arrives, their glossary is brought from the archive to the shelf. When the desk fills up, the least used glossary goes back to the archive.

And the most important part: the translator can have several glossaries open at once because they are working in parallel on five texts from five clients. It is not one glossary per document; it is one glossary per client, and client A’s documents use A’s glossary, client B’s use B’s, all on the same desk.

The analogy holds on five mappings:

The single translator = the base model (Llama-3-70B, Qwen2.5-72B). Expensive to train, a single copy in VRAM.
Each glossary = a LoRA adapter. Small (~150-400 MB), specific, easy to update.
The shelf with the glossaries within reach = the pool of adapters cached in VRAM (typically 50-200 at a time with an FP8 base on an H100 80 GB).
The archive = adapter storage on MinIO/S3/HF Hub. Hundreds or thousands, fetched on demand.
Working in parallel with several glossaries open = a heterogeneous batch with SGMV. The kernel does the grouped lookup into the right glossary for each word in the batch.

The bare mechanism: what a LoRA does and why it can be served multi-tenant

A LoRA adapter modifies a matrix W of the base model by adding a low-rank product to it:

$$W' = W + B A$$

where W ∈ R^{d_out × d_in} is the original matrix (the base weights), A ∈ R^{r × d_in} and B ∈ R^{d_out × r} are the adapter’s trainable matrices, and r is the rank (typically 8, 16, 32 or 64, always much smaller than d_in and d_out).

In a forward pass, instead of computing y = W' x, it computes:

$$y = W x + B(Ax)$$

That is: the base computation (Wx) happens exactly as before; the adapter adds two cheap matmuls (Ax and then B(·)) that supply the correction. The matrix BA is never explicitly materialised.

What this enables in serving: if you have the base loaded and N different adapters, Wx is computed only once for all the tokens in the batch (the base is the same). What changes between tokens is only the delta B_i(A_i x). If the tokens in the batch use different adapters, different deltas have to be applied per token, and that is what the SGMV kernel does in a single pass.

Without a specialised kernel this degenerates: you need to launch N separate matmuls (one per adapter), pay kernel launch overhead N times and lose the batching. With a specialised kernel (SGMV), every delta is computed in a single pass grouped by adapter.

SGMV: the kernel that holds it all up

SGMV (Segmented Gather Matrix-Vector multiplication) is the CUDA kernel Punica introduced and that vLLM, LoRAX, SGLang and TGI have adopted as their multi-LoRA engine.

Its job is to compute, given a batch of tokens with mixed adapters:

$$y_t = W x_t + B_{a(t)} A_{a(t)} x_t \quad \forall t \in \text{batch}$$

where a(t) is the adapter assigned to token t. SGMV operates in two phases:

SGMV-shrink: projection d_in → r with the corresponding A_{a(t)} matrix.
SGMV-expand: projection r → d_out with the corresponding B_{a(t)} matrix.

Internally, SGMV groups the batch’s tokens by adapter (it segments them), and for each segment it uses the optimal kernel for the size: for large segments (several requests with the same adapter) it goes through tensor cores; for small segments (one request per adapter) it uses the batched gather path that minimises launch overhead.

The result, in a single kernel pass, is the correct delta for every token in the batch, whatever its rank or its adapter. Punica reported up to 12× throughput vs vLLM/FasterTransformer/HF Transformers/DeepSpeed in heterogeneous multi-tenant scenarios; when every request uses the same adapter, SGMV is practically equivalent to the base without LoRA because it reduces to the “single large segment” case that is optimal for tensor cores.

S-LoRA refined SGMV with two kernels specific to different serving phases: MBGMM (Multi-size Batched Gather Matrix-Matrix) for prefill, MBGMV for decode. Both support different ranks among requests in the same batch, which was a limitation in the original SGMV.

The maths that matters

Three numbers drive every operational decision with multi-LoRA.

Memory per adapter. For a d_in × d_out matrix with rank r and b bytes per parameter (BF16/FP16 = 2):

$$\text{bytes per matrix} = (d_{\text{in}} \cdot r + r \cdot d_{\text{out}}) \cdot b$$

Summing over all the target matrices in each layer and multiplying by the number of layers gives the adapter size. A concrete calculation for Llama-3-70B, rank 16, BF16, all the matrices (Q, K, V, O, gate, up, down):

Matrix	Dimension	Bytes
Q (8192→8192)	8192·16 + 16·8192	524,288
O (8192→8192)	8192·16 + 16·8192	524,288
K (8192→1024)	8192·16 + 16·1024	294,912
V (8192→1024)	8192·16 + 16·1024	294,912
gate (8192→28,672)	8192·16 + 16·28,672	1,179,648
up (8192→28,672)	8192·16 + 16·28,672	1,179,648
down (28,672→8192)	28,672·16 + 16·8192	1,179,648
Sum per layer		5,177,344 ≈ 4.94 MB
Adapter total (80 layers)		~395 MB

If it is limited to attention-only (Q, O, V, K): about 125 MB per adapter. The choice of which matrices get LoRA belongs to whoever trains it; in serving it is inherited and it determines the cost.

Memory per rank. Linear: rank 8 → ~200 MB; rank 16 → ~400 MB; rank 32 → ~800 MB; rank 64 → ~1.6 GB. The simple sizing rule is: max_lora_rank should be the maximum rank you are going to serve, no more. Setting it higher wastes reserved memory in every slot.

How many adapters fit. For an H100 SXM 80 GB with a Llama-3-70B FP8 base (~70 GB), about 10 GB are left after a minimal KV cache → around 25 fully targeted r=16 adapters or around 80 attention-only. With an AWQ INT4 base (~35 GB), about 45 GB are left → hundreds of adapters. The rule: quantising the base does not just free memory, it multiplies the economics of the platform.

Latency overhead per adapter. Under real reported conditions:

Case	Typical overhead
All requests in the batch on the same adapter	~0 % (equivalent to a static merge)
Heterogeneous batch, similar ranks (e.g. all r=16)	10-30 % per layer
Heterogeneous batch, mismatched ranks (r=8 with r=128)	up to +84 % P95 TTFT for the lower rank
Naive PEFT LoRA (no SGMV)	250-950 % extra

It scales linearly with rank: rank 8 ≈ baseline; rank 64 ≈ 3-4 × overhead.

The real implementations in May 2026

Implementation	Base kernel	Hot-swap	Quantised base + LoRA	Operational notes
vLLM	SGMV + ext	Yes (LoRAResolver, S3/HF/FS plugins)	AWQ/GPTQ yes, bnb 4-bit offline only	The de facto default. `--enable-lora --max-loras N --max-lora-rank R --max-cpu-loras M`
LoRAX (Predibase)	Optimised SGMV	Yes (dynamic loading)	Yes	Designed specifically for multi-LoRA. Supports Medusa adapters per adapter (spec-dec per adapter).
SGLang	SGMV / csgmv	Yes	Yes	`--enable-lora-overlap-loading` cuts TTFT by up to 78 % on LoRA-heavy workloads
TensorRT-LLM	LoRA Executor (C++)	Pre-compile at build time	INT4 + LoRA common	Peak throughput on H100/B200, less flexible than vLLM
HF TGI	Punica fork	Yes (`LORA_ADAPTERS=...`)	Yes	In maintenance mode as of May 2026; HF recommends vLLM or SGLang
NVIDIA NIM	TRT-LLM under the hood	Static or dynamic (`NIM_PEFT_REFRESH_INTERVAL`)	Yes	Adapter store per model; polling for hot-add/remove

Three operational observations:

vLLM dominates open-source serving in 2026 thanks to the combination of a mature SGMV + LoRAResolver plugins + support for a quantised base. The critical parameter is --max-lora-rank: many installations set it to the maximum “just in case” and waste memory silently.
LoRAX wins in production operations with thousands of rarely used adapters thanks to its dynamic loading that does not block concurrent requests. A public case: Convirza with 60+ concurrent adapters and sub-2s P95.
SGLang wins on latency when cold starts are frequent thanks to --enable-lora-overlap-loading (async H2D during the compute of the previous request).

Combined pattern with quantisation and disaggregated serving

With quantisation (Quantisation). The canonical stack in May 2026 is base in FP8 (Hopper/Blackwell) or INT4 AWQ + adapters in BF16/FP16. The adapters are not quantised: they are small, the memory saved is irrelevant, and quantisation noise would accumulate badly with the delta. Quantising the base frees massive amounts of memory for more adapters with no significant loss (<1 % in typical MMLU with AWQ INT4, somewhat more in math/code/reasoning). An adapter trained with a BF16 base works with an FP8/INT4 base at inference with marginal loss, and that is what makes QLoRA operationally trivial: train with a 4-bit base and deploy with a consistent 4-bit base.

With disaggregated serving (Disaggregated serving). Multi-LoRA + prefill/decode disaggregation adds a management layer: each pod needs access to the request’s active adapter. The 2026 strategy: replicate hot adapters across every pod (prefill and decode), evict the cold ones. Cold adapters not present in the target pod are transferred on demand, accepting the extra cost in TTFT. Recent work (InfiniLoRA, FASTLIBRA, LoRAServe) automates this balancing, but the simple rule of thumb works in most deployments.

Operational pitfalls

Cold start. The first request to a sleeping adapter involves a fetch (from S3/MinIO/HF Hub) → CPU load → H2D copy. For an adapter of around 400 MB: 0.5-5 s typically, depending on bandwidth. Under high concurrency it can be up to 35 % of E2E latency (Predictive-LoRA paper, arXiv:2512.20210). Established mitigations: SGLang --enable-lora-overlap-loading (cuts TTFT by 35-78 %); vLLM pre-warming with a dummy request when an adapter is registered; predictive prefetching based on patterns (Predictive-LoRA cuts cold start by 68 %).

Heterogeneous worst case. If every request in the batch has a different adapter and a different rank, SGMV loses its advantage because each segment has size 1. Throughput can fall by up to 50 % vs the base without LoRA. Practical mitigation: group adapters by rank in the upstream routing, trying to get requests of the same rank into the same batch step.

Mismatched rank. Co-batching rank 8 with rank 128 penalises the smaller one: +84 % P95 TTFT for the rank-8 requests (Serving Heterogeneous LoRA Adapters paper, arXiv:2511.22880). Practice: normalise the rank across the fleet whenever possible (train every adapter at the same rank, or at least within a narrow range).

Eviction. LRU is the default. If you have more adapters than fit in CPU RAM, the evicted ones get fetched again from cold storage. Monitoring cold_starts_per_minute and cache_hit_ratio per endpoint is basic hygiene.

Base versioning. Every adapter is tied to an exact base version (Llama-3-70B-Instruct ≠ Llama-3.1-70B-Instruct). The routing must validate the base+adapter pair before serving, or the output will be silent rubbish.

Memory fragmentation. Without paged management (the pre-S-LoRA / pre-vLLM case), evicting and inserting adapters fragments HBM until it becomes unusable. Unified Paging solves it: the LoRA weights and the KV cache live in the same pool of blocks, interchangeable.

Implications on on-premise hardware

On an RTX 4090 (24 GB). The classic case: base Llama-3-8B FP8 (~8 GB) or Llama-3-8B AWQ-INT4 (~5 GB) + dozens of r=16 adapters (~80 MB each, around 3 MB per adapter for Llama-3-8B with r=16 attention-only). 50-100 active adapters fit easily on the 4090. It is the natural setup for multi-tenant demos, fine-tuning over an 8B base and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). This is where the serious setup comes in:

Llama-3-70B FP8 (~70 GB) fits on 2 H100s; the other 2 GPUs are available for batch + adapters.
Llama-3-70B AWQ-INT4 (~35 GB) fits on 1 H100; the rest of the cluster serves more concurrency or more adapters.
~200 fully targeted r=16 adapters fit with a comfortable budget on the cluster, enough for a SaaS platform with dozens of tenants and simultaneous A/B testing.
QLoRA training + consistent serving: the train→deploy adapter cycle takes hours, not days, because the adapter is around 400 MB instead of around 140 GB.

The rule of thumb on an H100 cluster in May 2026: an FP8 or INT4 base + 100-500 adapters per cluster is operationally trivial with vLLM or LoRAX; going past a thousand concurrent adapters starts to require serious tuning of eviction and prefetch.

Typical production stack

[API Gateway]
↓ (JWT with tenant_id / API key)
[Router]
↓ injects adapter_id into the request
[vLLM / LoRAX with --enable-lora]
--max-loras 16
--max-lora-rank 32
--max-cpu-loras 200
+ LoRAResolver → s3://adapters/{tenant_id}/{version}/
+ Base: Llama-3-70B-FP8 loaded once on 4×H100 TP=4
↓
[GPU]
FP8 base (70 GB) + ~150 BF16 adapters hot in HBM + ~1,000 warm in RAM
[MinIO / S3]
Cold storage for thousands of adapters
[CI pipeline]
→ trains new QLoRA adapter → push MinIO → notify server → warm-up
[Observability]
Prom: active_adapters, cache_hit_ratio, cold_starts_per_minute,
per_adapter_throughput, P99_with_lora vs P99_base

Routing pattern: Authorization: Bearer <key> → middleware extracts tenant_id → maps to adapter_id (Postgres or Redis) → POST /v1/completions with model: "<adapter_id>".

What we have not covered

DoRA (Weight-Decomposed LoRA): decomposes the update into magnitude + direction, closing part of the quality gap with full fine-tuning. Supported by TensorRT-LLM and others, but the serving pattern is identical to LoRA.
MoE + LoRA: how adapter fine-tuning is done over an MoE, what happens to the routing. Not trivial, an active research area in 2026.
Activated LoRA (arXiv:2512.17910): a variant that reuses the KV cache between compatible adapters, cutting the cost of cold start with shared prefixes.
LoRA for speculative decoding: each adapter brings its own Medusa head, supported by LoRAX as “Turbo LoRA”.
Compress-then-Serve (arXiv:2407.00066): quantising the adapters themselves to serve even more concurrently. Still a marginal practice in production as of May 2026.

References

Hu, E. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Dettmers, T. et al. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Brüel-Gabrielsson, R. et al. Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead. 2024. https://arxiv.org/abs/2407.00066
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference. 2025. https://arxiv.org/abs/2511.22880
InfiniLoRA: Disaggregated Multi-LoRA Serving. 2026. https://arxiv.org/abs/2604.07173
Predictive-LoRA: Proactive Fragmentation-Aware Serverless LoRA Serving. https://arxiv.org/abs/2512.20210
S-LoRA repo: https://github.com/S-LoRA/S-LoRA
Punica repo: https://github.com/punica-ai/punica
LoRAX repo (Predibase): https://github.com/predibase/lorax
vLLM LoRA docs: https://docs.vllm.ai/en/stable/features/lora/
SGLang LoRA docs: https://docs.sglang.io/advanced_features/lora.html
TensorRT-LLM LoRA docs: https://nvidia.github.io/TensorRT-LLM/advanced/lora.html
NVIDIA NIM PEFT: https://docs.nvidia.com/nim/large-language-models/latest/peft.html
HF TGI Multi-LoRA blog: https://huggingface.co/blog/multi-lora-serving
LMSYS S-LoRA recipe blog: https://www.lmsys.org/blog/2023-11-15-slora/
Predibase LoRAX blog: https://predibase.com/blog/lorax-the-open-source-framework-for-serving-100s-of-fine-tuned-llms-in

Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

Wed, 27 May 2026 08:00:00 +0200

This post is the natural continuation of continuous fine-tuning in production, which covers the operational cycle (Postgres + SQL queries + hot-swap) that feeds these methods with real data. Here we go inside each one: what each loss optimises, what hypothesis it makes, and why they choose one or the other.

TL;DR

Classic RLHF, the kind in the InstructGPT papers, is practically extinct in production. The reason is not ideological: it is that in 2023 Rafailov and others showed that the reward model does not need to exist as a separate object. With an elegant change of variable, the optimal policy can be trained directly from preference pairs, without going through a reward and without RL. That is DPO. From there the family has branched into four methods that coexist in 2026: DPO when you have (chosen, rejected) pairs, KTO when you only have a binary 👍/👎 signal, ORPO when you want SFT and preferences in the same pass to save memory, and SimPO when you also want to get rid of the reference model and normalise by length. This post explains exactly what each loss does, proves it with an end-to-end numerical example, gives the real decision table between the four and takes apart the three biases that kill the method in production when nobody watches them.

You are here: TUNE

The six-stage LLMOps pipeline places Tune between Data and Eval. Inside Tune three modalities coexist: SFT (supervised fine-tuning), preference optimization (what this post covers) and agent training / RFT. What follows is the inside of the second modality.

The analogy: the sommelier who trains the palate without a theory book

Imagine you want to train a sommelier. You have two routes. The first is teaching them oenological theory: grape varieties, terroirs, vinification methods, barrel types. You give them cards with canonical wine descriptions and ask them to memorise them. That is SFT: (prompt, ideal answer) pairs. It works as general education, but the sommelier who comes out of it cannot tell two excellent Riojas apart.

The second route is the blind comparator. You put two identical opaque glasses in front of them. You say “this one is better than this one”. You do not explain why. You repeat the exercise a thousand times, with a thousand different pairs. After a while, the sommelier does not need you to say it: they have a trained palate. They have not learned oenological theory, they have learned to discriminate.

DPO, KTO, ORPO and SimPO are four variants of the second route. All four train the model to discriminate, not to memorise. The differences between them are:

DPO: two glasses at each tasting, the sommelier always knows which one is the “good” one and which is the “bad” one.
KTO: a single glass each time, they are only told “you would like this” or “you would not like this”. No pairs.
ORPO: the tasting also includes a small embedded theory class (SFT in parallel) so the catalogue is not forgotten.
SimPO: like DPO, but the sommelier does not compare against “what your master would have said” (reference model); they compare the two glasses directly, normalising by the amount of liquid in each.

The analogy is not decorative: the rest of the post is that same idea expressed mathematically.

Why DPO exists: Rafailov’s trick

To understand DPO it helps to walk through in thirty seconds what it replaces. Classic RLHF, the InstructGPT kind, has three phases:

Phase 1 – SFT. You train the model on (prompt, ideal answer). Out comes π_ref: the reference policy. It is the “educated” model.

Phase 2 – Reward model. On the same model (a different head), you train a regressor: given (prompt, chosen, rejected), it learns to give a higher score to the chosen. Out comes r_φ(x, y).

Phase 3 – PPO. You take π_ref as the starting point and train another copy π_θ so that it maximises r_φ(x, π_θ(x)) with a KL penalty to keep it from drifting too far from π_ref. That requires generating rollouts (on-policy decode at every step), keeping the three models in memory (π_θ, π_ref, r_φ), and a classic RL setup: unstable, sensitive to hyperparameters and famous for its reproducibility problems.

Rafailov’s observation was: phase 3 has a closed-form solution. If you write out the exact optimisation problem PPO solves

$$\max_{\pi_\theta} \; \mathbb{E}_{x \sim D, \, y \sim \pi_\theta(\cdot|x)} \big[ r_\phi(x,y) \big] - \beta \, \mathrm{KL}\big(\pi_\theta(\cdot|x) \,\|\, \pi_\mathrm{ref}(\cdot|x)\big)$$

it turns out the optimal policy has the form

$$\pi^{*}(y|x) = \frac{1}{Z(x)} \, \pi_\mathrm{ref}(y|x) \, \exp\!\left( \tfrac{1}{\beta} r_\phi(x,y) \right)$$

and from there you can solve for the implicit reward:

$$r_\phi(x,y) = \beta \log \frac{\pi^{*}(y|x)}{\pi_\mathrm{ref}(y|x)} + \beta \log Z(x).$$

The second term is a function of x only and cancels out when you compare two answers to the same prompt. The reward does not need to be learned: it is implicit in the log-prob ratio between the trained model and the reference one. If you plug that into the Bradley-Terry model for preferences (the formula that says “the probability that yw is preferred to yl is σ(r(x,yw) - r(x,yl))”), out comes the DPO loss:

$$\mathcal{L}_\mathrm{DPO}(\pi_\theta;\pi_\mathrm{ref}) = -\,\mathbb{E}_{(x, y_w, y_l) \sim D} \log \sigma\!\Big( \beta \big[ \log \tfrac{\pi_\theta(y_w|x)}{\pi_\mathrm{ref}(y_w|x)} - \log \tfrac{\pi_\theta(y_l|x)}{\pi_\mathrm{ref}(y_l|x)} \big] \Big).$$

That is the whole of DPO. There is no reward model, no RL, no rollouts. Only log-probs over static data.

DPO with real numbers

The formula is less intimidating when you evaluate it with an example. Imagine a pair from the dataset:

x = “Explain what a KVM switch is”.
y_w = correct answer (chosen).
y_l = confusing answer (rejected).

After a forward pass we have four numbers (sums of per-token log-probs, negative sign because each log p_token ≤ 0):

Quantity	Value
`log π_θ(y_w \| x)`	−45.2
`log π_θ(y_l \| x)`	−52.1
`log π_ref(y_w \| x)`	−47.3
`log π_ref(y_l \| x)`	−50.8

The “ratios” are the improvement of the trained model over the reference one for each answer:

For y_w: −45.2 − (−47.3) = +2.1 → the trained model gives it more probability than the reference one. Good.
For y_l: −52.1 − (−50.8) = −1.3 → the trained model gives it less probability. Also good.

With β = 0.1 (a typical value) the “margin” inside the logarithm is:

$$m = \beta \cdot (2.1 - (-1.3)) = 0.1 \cdot 3.4 = 0.34$$

And the loss:

$$\mathcal{L}_\mathrm{DPO} = -\log \sigma(0.34) = -\log(0.584) \approx 0.538.$$

The intuition is visible straight away: the more the chosen log-prob rises and the more the rejected one falls (relative to π_ref), the more positive m becomes, the higher the sigmoid and the lower the loss. If the margin is negative (the model gets it wrong), σ(m) < 0.5 and the loss blows up. The gradient pushes the model to increase π_θ(y_w|x) and lower π_θ(y_l|x).

The role of β: it is the temperature of the alignment. If β is small, the model is allowed to drift a long way from π_ref; if it is large, the KL weighs heavily and the model barely moves. Typical value in 2026: 0.05–0.3, with 0.1 as a starting point.

KTO: when you only have 👍/👎

DPO needs pairs. In practice, that is almost never what the product gives you: what the product gives you is binary feedback (a thumbs up or a thumbs down, a conversion or an abandonment). KTO — Kahneman-Tversky Optimization, Ethayarajh et al. 2024 — solves exactly that case.

The intuition comes from Kahneman and Tversky’s prospect theory: humans are more sensitive to losses than to gains of the same magnitude (loss aversion: losing 100 € hurts more than gaining 100 €). KTO carries that over into the loss:

$$\mathcal{L}_\mathrm{KTO}(x, y) = \begin{cases} \lambda_d \big[ 1 - \sigma\!\big(\beta \cdot ( h_\theta(x,y) - z_0 ) \big) \big] & \text{if } y \text{ is desirable},\\ \lambda_u \big[ 1 - \sigma\!\big(\beta \cdot ( z_0 - h_\theta(x,y) ) \big) \big] & \text{if } y \text{ is undesirable}, \end{cases}$$

where h_θ(x,y) = log(π_θ(y|x) / π_ref(y|x)) is exactly the same ratio that appeared in DPO, z_0 is an estimate of the batch KL divergence (it acts as a “neutral point”) and λ_d, λ_u are the desirable / undesirable weights.

The critical point: KTO does not need pairs. Each example is (prompt, answer, binary label). This fits real product telemetry. The usual practical rule is λ_u > λ_d (e.g. 1.0 vs 0.33) when the database has more 👍 than 👎, so that the negative signal is not diluted.

KTO works particularly well in two scenarios:

Products with explicit feedback UX (chatbots with 👍/👎): each thumbs is a direct KTO example, no need to synthesise pairs.
Imbalanced datasets (far more 👍 than 👎, or the other way round): the weights λ_d, λ_u handle it explicitly.

ORPO: SFT and preferences in a single pass

DPO assumes you have already done SFT. You train two phases: first SFT to get π_ref, then DPO on top of π_ref. Two passes over the data, two optimisations, two models in memory.

ORPO — Odds Ratio Preference Optimization, Hong et al. 2024 — merges both phases. The loss combines two terms:

$$\mathcal{L}_\mathrm{ORPO} = \mathcal{L}_\mathrm{SFT}(y_w) + \lambda \cdot \mathcal{L}_\mathrm{OR}(y_w, y_l)$$

The first term is classic SFT on the chosen answer (negative cross-entropy). The second is the odds ratio between chosen and rejected:

$$\mathcal{L}_\mathrm{OR} = -\log \sigma\!\Big( \log \tfrac{\mathrm{odds}_\theta(y_w|x)}{\mathrm{odds}_\theta(y_l|x)} \Big), \quad \text{with } \mathrm{odds}_\theta(y|x) = \tfrac{P_\theta(y|x)}{1 - P_\theta(y|x)}.$$

What matters: there is no π_ref. ORPO trains a single model, in a single pass, without loading the reference policy into memory. On paper it sounds good and in practice it works: a Llama 3 8B aligned with ORPO over 5k pairs takes about 3 hours on 4×H100 and fits in VRAM with aggressive QLoRA on a single RTX 4090.

The λ parameter is the weight of the preference term. Typical: 0.1–0.3. If λ is too high, the model learns to discriminate but forgets the SFT (catastrophic forgetting); if it is too low, the alignment barely shows.

SimPO: do you really need a reference model?

SimPO — Simple Preference Optimization, Meng et al. 2024 — takes ORPO’s question one step further: if ORPO frees itself from π_ref for the combined SFT+preference case, why not free yourself from π_ref in the pure DPO case too?

The SimPO loss:

$$\mathcal{L}_\mathrm{SimPO} = -\log \sigma\!\Big( \tfrac{\beta}{|y_w|} \log \pi_\theta(y_w|x) - \tfrac{\beta}{|y_l|} \log \pi_\theta(y_l|x) - \gamma \Big).$$

Two changes with respect to DPO:

There is no π_ref: absolute log-probs of the trained model are compared directly.
Length-normalization: each log-prob is divided by the length of its answer |y|. This is key because without normalising, long answers tend to have a lower total log-prob (each token contributes its log p < 0), creating an artificial bias.
Explicit margin γ: the loss is low if the difference of normalised log-probs exceeds γ. Typical: γ = 0.5–1.5.

With β = 2.0, γ = 1.0, |y_w| = 120 tokens, |y_l| = 100 tokens and the earlier log-probs:

$$m = \tfrac{2.0}{120} \cdot (-45.2) - \tfrac{2.0}{100} \cdot (-52.1) - 1.0 = -0.753 + 1.042 - 1.0 = -0.711.$$ $$\mathcal{L}_\mathrm{SimPO} = -\log \sigma(-0.711) \approx 1.11.$$

A higher loss than DPO at the same point: SimPO takes longer to converge, but uses half the memory (a single model in VRAM) and removes the dependency on π_ref. It is the dominant option when memory is the bottleneck.

Decision table: which one to use and when

Signal available	Memory available	Prior SFT	Recommended method	Rationale
`(chosen, rejected)` pairs	High (≥ 80 GB GPU)	Yes	DPO	Most established baseline, better reproducibility
`(chosen, rejected)` pairs	Low (24–48 GB GPU)	Yes	SimPO	Removes `π_ref` → ~50 % less VRAM
`(chosen, rejected)` pairs	Any	No	ORPO	SFT and preferences in one pass
Binary 👍/👎 signal without pairs	High	Yes	KTO	The only method native to unpaired data
Binary signal + few pairs	High	Yes	KTO with a DPO sub-batch	Combination documented in TRL 0.13
Multistep trajectories (tool use)	Very high	Yes	Pure RLHF/RFT	Preference-pair methods do not capture the dynamics

Typical dataset magnitudes:

Method	Minimum viable	Sweet spot	Plateau
DPO	1,000 pairs	5,000–20,000	> 50,000
SimPO	2,000 pairs	5,000–20,000	> 50,000
ORPO	3,000 pairs (includes SFT)	10,000–30,000	> 80,000
KTO	5,000 binary examples	20,000–80,000	> 200,000

KTO typically needs 3-4× more volume than DPO because the binary signal is weaker than the comparative one. The trade-off is that the binary signal is the one a production product naturally produces.

The three biases that break the method

All four methods share one problem: they are training on proxies for quality, not on quality. Those proxies have systematic biases the model can exploit trivially.

Length bias

Documented in the original DPO paper and in the literature that followed. Long answers tend to be preferred by humans, probably because they look “more complete”. If the pair dataset inherits that bias, the model learns that lengthening the answer is what gets rewarded, not that better content is what gets rewarded. Result: after 2–3 epochs the model spews waffle.

Mitigations:

DPO: filter the dataset, removing pairs where |y_w| > 1.3 · |y_l| (the 30 % rule).
SimPO: the length-normalization in the loss fixes it by construction.
ORPO / KTO: dataset filtering or an auxiliary length regularisation (DPOP, R-DPO).

The chart shows the usual pattern: with no mitigation, in 3 epochs a Llama 3 8B with DPO can go from answers of about 150 tokens to answers of about 280 tokens, without any improvement in human-evaluated quality. SimPO keeps the length roughly stable.

Position bias (in dataset curation)

If the pairs are generated automatically with an LLM judge (covered in detail in the next post of this series), there is a known bias: judges prefer the first answer they see when the two are comparable. If every pair in the dataset always has the chosen in position A and the rejected in position B, the model does not learn preference: it learns an artefact of the curation process.

Mitigation: random shuffling of the order in the query to the judge and averaging two passes with the order reversed (seen in tools such as Promptfoo and Inspect AI by default).

Distribution shift between the data and `π_ref`

DPO, KTO and SimPO implicitly compare the trained model against a distribution. If the preference data comes from a model (another LLM generating candidates) very different from π_ref, the trained model can explore regions where π_ref has almost zero probability, giving numerically unstable ratios (the log of very small quantities). In practice this shows up as loss explosions, NaN gradients or silent regression.

Mitigation: generate the dataset candidates with π_ref itself whenever possible (rejection sampling over π_ref, with an external judge picking the chosen). That is the canonical on-policy RLHF prescription applied to the offline setting.

Implications on on-premise hardware

The figures below are indicative for a typical May 2026 scenario: Llama 3.1 8B Instruct as π_ref, a dataset of 5,000–20,000 pairs, QLoRA (NF4) with LoRA rank 16 over all the transformer block projectors (q,k,v,o,gate,up,down), effective batch size 16, 1–3 epochs.

On an RTX 4090 (24 GB)

Method	Peak VRAM	Time / epoch (5k pairs)	Notes
DPO	~22 GB	50–80 min	Needs `π_ref` in VRAM even if quantised FP8/INT8
SimPO	~14 GB	45–70 min	No `π_ref`, the natural option on a 4090
ORPO	~16 GB	60–90 min	No `π_ref`, comfortably viable
KTO	~22 GB	90–150 min (10k binary)	Same VRAM as DPO, more data per epoch

The 4090 (24 GB, Ada Lovelace, no NVLink) is perfectly viable for Llama 8B with QLoRA if you pick the method sensibly. For 13B the choice between SimPO/ORPO is no longer a preference, it is a requirement.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Method	Viable model	Time / epoch (10k pairs)	Notes
DPO	Llama 3 70B (4-bit)	60–90 min	Tensor parallel = 4, still comfortable
SimPO	Llama 3 70B (BF16)	50–75 min	Full BF16 fits thanks to having no `π_ref`
ORPO	Llama 3 70B (BF16)	70–100 min	Similar to SimPO in consumption
KTO	Llama 3 70B (4-bit)	100–140 min	Larger datasets offset by parallelism

On an NVLink cluster the operational difference between methods blurs: they all fit. The choice goes back to what kind of signal you have, not to budget.

What we have not covered (upcoming articles)

LoRA and QLoRA fundamentals: the maths of the low-rank adapter that underpins everything above, and why a 70B fits in 24 GB.
LLM-as-judge fundamentals: how to build the judge that generates the chosen/rejected pairs without position or verbosity bias. The next post in this batch covers it.
Online DPO and iterative on-policy: the 2026 research state of the art (Fast-Slow Chasing, RLOO, iterative preference learning) and why it is not production yet.
Distillation and synthetic preference data: when it is worth generating pairs with a large model to train a small one.

References

Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. KTO: Model Alignment as Prospect Theoretic Optimization (ICML 2024).
Hong, J., Lee, N., Thorne, J. ORPO: Monolithic Preference Optimization without Reference Model (EMNLP 2024).
Meng, Y., Xia, M., Chen, D. SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
HuggingFace TRL 0.13 — reference implementations: https://huggingface.co/docs/trl.
Tunstall, L. et al. The Alignment Handbook — reproducible recipes: https://github.com/huggingface/alignment-handbook.
Park, R. et al. Disentangling Length from Quality in Direct Preference Optimization (R-DPO, ACL 2024).

Continuous fine-tuning in production: from real traffic to the deployed adapter

Thu, 21 May 2026 10:00:00 +0200

TL;DR

Continuous fine-tuning is not “training the model every so often”. It is a closed loop where real production traffic generates the datasets, a short pipeline trains a LoRA adapter, a battery of evaluations decides whether it is promoted, and vLLM loads it without restarting. The state of the art in May 2026 has fragmented the stack: it is no longer DPO against everything, but a choice between SFT, DPO, KTO, ORPO and SimPO according to the type of signal your product captures. What has consolidated the pattern is the combination of PostgreSQL 18 + pgvector 0.8 as the nervous system of the pipeline, traffic capture, dataset versioning, eval results, adapter registry, together with vLLM multi-LoRA hot-swap, which turns deployment into an HTTP call. This article takes the cycle apart with concrete schemas, real queries, and the numbers it costs on an RTX 4090 against a 4×H100 cluster.

You are here: Tune + Retrain

This post crosses two stages of the six-stage LLMOps pipeline: the decision to train a new adapter (the Tune stage) is triggered by the signals from Observe that travel through the Retrain stage until the loop closes. The post takes apart the complete circuit between the two boxes.

Picture a neighbourhood restaurant with a signature dish that works, but the chef knows it can be tuned. Every night things happen:

Some diners leave part of the dish: a weak signal that something did not quite land.
Others ask for another version (“could you put less salt in it?”): an explicit, directional signal.
Others finish the dish and come back the following week: the only signal that really matters, but it arrives late.
And a select group offers an opinion without being asked, usually a negative one.

The chef does not redo the menu every night. He does something more interesting: he writes down in a notebook the dishes served, the returns, the changes requested, the tips. Every so often, he reads the whole notebook, decides on minimal adjustments to a recipe, tries the new version at a private table with his staff, and only if they take to it does he add it to the next day’s menu. Sometimes he even serves two different versions of the dish to different tables for a week, measures what happens, and picks one.

That is continuous fine-tuning. The notebook is Postgres. The dish is the base model. The notes are feedback signals, explicit and implicit. The “minimal adjustment” is a 30 MB LoRA adapter. The private table is the battery of automated evaluations. The next day’s menu is vLLM with multi-LoRA hot-swap, which loads the new adapter without restarting the service. Serving two versions to different tables is A/B testing with real traffic.

The analogy is exact on one critical point: the chef does not throw away the original recipe. He keeps the base recipe and holds a separate notebook with the “modifications that work well for the neighbourhood regulars”. That notebook is the LoRA adapter: on top of the base model, not in its place.

The cycle, taken apart

Before going into components, it helps to fix the complete flow. These seven steps are what any serious team replicates with variations:

The cycle lasts between 1 and 4 weeks in real production. What changes between teams is the tempo (faster in assistant chat, slower in regulated banking) and the details of each step. The structure is the same.

Why continuous fine-tuning (and why it is not RAG)

Before going deeper, a distinction that people still confuse. Fine-tuning is for form, not for facts. If your problem is that the model does not know the customer’s rates or the updated catalogue, do not fine-tune: use RAG. If your problem is that the model responds in a tone that does not fit, does not respect your JSON format, refuses legitimate cases or invents structure, then fine-tuning is the answer.

In 2026 the boundary is well established by community practice:

Observed problem	Solution
The model does not know X (X changes weekly)	RAG
The model knows X but responds badly in tone or format	SFT fine-tuning
There are two ways of responding and I prefer one over the other	Fine-tuning with preferences (DPO/KTO/ORPO/SimPO)
The model reasons badly in a verifiable domain (code, maths)	RL with verifiable reward (GRPO/DAPO)
The model is competent, it just needs a memory of facts	RAG, not fine-tuning

Continuous fine-tuning is the disciplined version of the second and third cases. The key word is continuous: it is not a one-off “we aligned the model” event, it is a process that runs every time the traffic distribution drifts far enough, or new use cases appear.

The four techniques according to the signal you capture

The most important change of the last 12 months has been the end of DPO’s monopoly. In 2024 every team doing alignment used DPO with (chosen, rejected) pairs. In 2026 the choice is finer and depends on what the signal you collect in your product looks like:

Real signal in the product	Recommended technique	Why
Labelled correct examples (input → expected output)	SFT + LoRA	Still the baseline. 500-5,000 examples are enough for style.
Explicit `(chosen, rejected)` pairs	DPO or SimPO	SimPO removes the reference model → 50 % less VRAM in training.
Loose 👍 / 👎 on responses	KTO	The method that fits real telemetry most naturally.
SFT and preferences in a single pass	ORPO	A single model in memory, avoids drift between phases.
Verifiable reward (tests, solutions)	GRPO / DAPO	Reasoning, not chat. A different world.

The practical rule: design the feedback capture in the product thinking about which method you will be able to use afterwards. If your UI only has 👍/👎, you force the path to KTO. If you add a “regenerate response” button, you unlock DPO from regenerate-as-rejected (we will see it below). If you add an “edit response” button, the edited response becomes high-quality direct SFT.

There is a cost detail that gets little publicity. DPO needs to keep two models in memory: the one you are training and the reference one. SimPO removes that second model. ORPO does too. For a Llama 3 8B in BF16 this is the difference between needing ~32 GB of active VRAM during training (DPO) or ~16 GB (SimPO/ORPO). It is the difference between the training fitting on an RTX 4090 with aggressive QLoRA, or not fitting without offload.

Postgres as the nervous system of the pipeline

Here is the strong technical opinion of this article, and it is one worth defending with data: Postgres 18 + pgvector 0.8 + an S3/MinIO bucket for the weights is enough for the whole pipeline. You do not need MLflow, you do not need lakeFS, you do not need DVC.

This is not ideological minimalism. It is about three concrete advantages that no alternative stack matches in the on-premise scenario with compliance:

A single source of truth, a single authorisation model. The ACLs you already have for Postgres cover the training data, the eval results, the adapter registry and the audit log. You do not multiply control planes.
SQL as the pipeline’s universal language. The query that generates the dataset, the eval gate predicate, the A/B traffic assignment, the promotion decision: it is all SQL. Your team already knows SQL.
Audit and cryptographic reproducibility for free. The pg_audit and pgcrypto extensions, combined with set_hash over the dataset, give you cryptographic traceability with no additional code. It is a subject that would fill an article of its own.

A concrete schema

We start with the traffic table, partitioned by week so that DROP PARTITION is cheap:

CREATE TABLE obs.inference_log (
 id BIGSERIAL,
 request_id UUID NOT NULL,
 tenant_id INT NOT NULL,
 user_hash BYTEA, -- GDPR pseudonymisation
 adapter_id TEXT NOT NULL, -- e.g. "support-es-v4.1"
 experiment TEXT, -- e.g. "rerank-v2-canary"
 variant CHAR(1), -- 'A' | 'B' | NULL
 messages JSONB NOT NULL,
 completion TEXT,
 ttft_ms INT,
 tokens_in INT,
 tokens_out INT,
 -- feedback signals
 fb_explicit SMALLINT, -- -1/0/+1 (KTO-ready)
 fb_regen BOOLEAN DEFAULT false, -- user regenerated -> DPO-rejected
 fb_edited BOOLEAN DEFAULT false, -- user edited -> SFT golden
 parent_id BIGINT, -- self-reference for regenerate
 -- vector and meta
 embedding HALFVEC(1024), -- pgvector 0.8, half the RAM
 pii_flags SMALLINT DEFAULT 0, -- bitmask
 created_at TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);

CREATE TABLE obs.inference_log_2026w21 PARTITION OF obs.inference_log
 FOR VALUES FROM ('2026-05-18') TO ('2026-05-25');

CREATE INDEX ON obs.inference_log_2026w21
 USING hnsw (embedding halfvec_cosine_ops);
CREATE INDEX ON obs.inference_log_2026w21
 (tenant_id, adapter_id, created_at);

Three decisions deserve a note:

HALFVEC(1024). Native FP16 vectors in pgvector 0.8. Half the RAM and disk, with a loss of precision that is irrelevant for semantic deduplication. This alone, at a scale of millions of rows, saves between 4 and 8 GB.

Weekly partitioning by time range. At 90 days, DROP TABLE obs.inference_log_2026wXX frees space in milliseconds without a prolonged lock. Autovacuum never touches frozen partitions again.

Self-referencing parent_id. The user regenerates the response → a new row is inserted with parent_id pointing at the previous one. That will give us a DPO dataset without touching the UX.

The adapter registry

CREATE TABLE serve.adapter (
 id TEXT PRIMARY KEY,
 base_model TEXT NOT NULL,
 rank INT, alpha INT,
 target_modules JSONB,
 method TEXT, -- 'sft'|'dpo'|'kto'|'orpo'|'simpo'
 training_run_id UUID,
 dataset_snapshot_id UUID,
 weights_uri TEXT, -- s3://.../v4.2.safetensors
 eval_summary JSONB,
 status TEXT NOT NULL, -- 'training'|'canary'|'prod'|'retired'
 traffic_pct NUMERIC(5,2) DEFAULT 0,
 promoted_at TIMESTAMPTZ
);

The vLLM router reads this table with a TTL of a few seconds. An UPDATE serve.adapter SET status='prod', traffic_pct=100 WHERE id='v4.2' is a promotion. An UPDATE ... SET status='retired' is a rollback. The audit of who did what and when comes from pg_audit without writing a single additional line of code.

Generating DPO and KTO datasets from real traffic

This is where the elegance of the schema pays off. The dataset is not a static file: it is a materialised view built with SQL over obs.inference_log.

KTO dataset from 👍/👎

KTO is the method that best fits the signal any decent chat product captures. The query:

CREATE MATERIALIZED VIEW train.kto_v3_candidate AS
SELECT
 messages AS prompt,
 completion AS response,
 CASE WHEN fb_explicit > 0 THEN true ELSE false END AS label,
 adapter_id, created_at
FROM obs.inference_log
WHERE fb_explicit != 0
 AND created_at > now() - interval '60 days'
 AND pii_flags = 0
 AND tenant_id IN (SELECT id FROM tenant WHERE consent_training);

Simple. Every row with explicit feedback becomes an example (prompt, response, desirable_yes_no). KTO trains directly on this signal, with no need to build pairs.

DPO dataset from “regenerate”

The trick that is worth this article on its own. When the user presses “regenerate response”, they are giving an extraordinarily strong signal: the first response was no good to them. If the second is neither regenerated nor rated negatively, we assume it was. That is a DPO pair without a single extra click in the UI:

CREATE MATERIALIZED VIEW train.dpo_v3_candidate AS
SELECT
 rej.messages AS prompt,
 cho.completion AS chosen,
 rej.completion AS rejected,
 rej.adapter_id
FROM obs.inference_log rej
JOIN obs.inference_log cho ON cho.parent_id = rej.id
WHERE rej.fb_regen = true
 AND cho.fb_explicit >= 0
 -- length bias mitigation in classic DPO
 AND cho.tokens_out BETWEEN rej.tokens_out * 0.5
 AND rej.tokens_out * 2.5;

The clause on lengths is the cheap cure for the length bias documented in DPO. Without it, the model learns that “longer = better” because the responses the user accepts tend to be slightly longer. With SimPO or ORPO this filter is optional; with classic DPO it is necessary.

Semantic deduplication with pgvector

Before training, dedup. Two almost identical prompts in the dataset are noise that biases the model:

WITH ranked AS (
 SELECT id, embedding,
 row_number() OVER (
 PARTITION BY hashtext(messages::text)
 ORDER BY fb_explicit DESC, created_at DESC
 ) AS rn
 FROM obs.inference_log
 WHERE created_at > now() - interval '60 days'
)
DELETE FROM train.kto_v3_candidate kto
USING ranked r
WHERE r.rn > 1 AND kto.id = r.id;

And for the semantic duplicates (paraphrases) we use pgvector 0.8 directly with an iterative index scan:

-- Find near-duplicates of any given example
SELECT id, messages, embedding <=> $1 AS dist
FROM obs.inference_log
WHERE created_at > now() - interval '60 days'
 AND embedding <=> $1 < 0.05
ORDER BY embedding <=> $1
LIMIT 50;

The iterative scan is a key improvement in pgvector 0.8: before, the HNSW index could return fewer results than requested when there were additional filters (WHERE); now it iterates until it meets the limit. Without that improvement, curation queries over datasets of millions of rows were unworkable without a brutal pre-filter.

Eval gates: three stages, all SQL

The most common mistake when implementing continuous fine-tuning is skipping or watering down the eval gates. That turns the cycle into a roulette wheel. The pattern that works in 2026 is three stages, each with a different latency/coverage trade-off:

And this is where Postgres shines again: the promotion gate is expressed as an SQL predicate. Nothing more:

CREATE TABLE train.eval_result (
 adapter_id TEXT REFERENCES serve.adapter(id),
 suite_id TEXT, -- 'safety-es', 'support-helpfulness'
 metric TEXT,
 score NUMERIC,
 judge_model TEXT,
 judged_at TIMESTAMPTZ DEFAULT now(),
 PRIMARY KEY (adapter_id, suite_id, metric)
);

CREATE OR REPLACE FUNCTION serve.can_promote(candidate TEXT, current TEXT)
RETURNS BOOLEAN AS $$
 SELECT NOT EXISTS (
 SELECT 1
 FROM train.eval_result c
 JOIN train.eval_result p USING (suite_id, metric)
 WHERE c.adapter_id = candidate
 AND p.adapter_id = current
 AND suite_id IN ('safety-es','support-helpfulness','refusal-rate')
 AND c.score < p.score * 0.98 -- 2 % tolerance
 );
$$ LANGUAGE sql STABLE;

An SQL function as the gate. Usable from CI with psql -c "SELECT serve.can_promote('v4.2','v4.1')" and an exit code of 0/1. No orchestrator needed, no specific UI needed. The audit stays in the Postgres log.

vLLM multi-LoRA: the deploy is an HTTP POST

Two years ago, deploying a new fine-tune meant rotating inference pods. Today it is an HTTP call. vLLM 0.7+ supports loading and unloading LoRA adapters hot, keeping several resident in VRAM and choosing the right one per request.

Server configuration:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --enable-lora \
 --max-loras 4 \
 --max-lora-rank 64 \
 --env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Deploying a new adapter:

curl -X POST http://localhost:8000/v1/load_lora_adapter \
 -H "Content-Type: application/json" \
 -d '{
 "lora_name": "support-es-v4.2",
 "lora_path": "/mnt/adapters/support-es-v4.2"
 }'

From that moment on, requests including "model": "support-es-v4.2" are served with that adapter applied on top of the base model. Switching between adapters has negligible latency (the most recent research on Activated LoRA takes this to levels where the switching cost is invisible).

This changes operations substantially. Deploying a new fine-tune stops being an infrastructure event and becomes a state change in Postgres. The router queries the serve.adapter table, sees that v4.2 is in canary with traffic_pct=5, and directs 5 % of requests to the new adapter. The exact route of that 5 % is decided with deterministic hashing of the user_id so that the same user always sees the same variant (sticky):

-- No assignment table, no additional state
-- the variant is computed in-place in SQL or in the router:
SELECT
 CASE WHEN hashtext($user_id || $experiment) % 100
 < (SELECT traffic_pct FROM serve.adapter WHERE id = $candidate)
 THEN $candidate ELSE $current END AS adapter_id;

A/B with real traffic: measure or live deceived

The eval gates measure against fixed benchmarks. That is necessary but insufficient. Reality is only measured with real traffic. Once the adapter is in canary, what matters are the online metrics measured over obs.inference_log for each variant:

SELECT
 adapter_id,
 COUNT(*) AS n,
 AVG(fb_explicit) AS mean_score,
 STDDEV(fb_explicit) / SQRT(COUNT(*)) AS sem,
 AVG(ttft_ms) AS ttft_avg,
 percentile_cont(0.5) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p50,
 percentile_cont(0.95) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p95,
 AVG(CASE WHEN fb_regen THEN 1 ELSE 0 END) AS regen_rate
FROM obs.inference_log
WHERE experiment = $1
 AND created_at > now() - interval '7 days'
GROUP BY adapter_id;

What to look at: explicit feedback, latency (TTFT, p50, p95), regeneration rate. An adapter that raises the mean feedback but also raises the regeneration rate is suspicious, probably it is responding in a flashier but less useful way. An adapter that lowers latency but lowers feedback may be worth studying: it may be being more concise than it should.

Promotion to prod happens when, after 24-72 hours in canary, the candidate adapter beats the current one on at least one key metric without degrading the others. Again: it is an UPDATE in Postgres.

Applied to typical on-premise hardware

Let us come down to two representative configurations, one for iteration and one for production.

Case 1 — RTX 4090 (24 GB) for development iteration

An RTX 4090 with 4-bit QLoRA can train adapters on an 8B model without trouble. The VRAM budget combines four components; the KV cache during the intermediate evaluations is not negligible and deserves an explicit margin:

8B base model in 4-bit: ~5 GB
Activations + gradients: ~8 GB (depends on batch and context)
Optimizer state (LoRA r=16): ~0.5 GB
KV cache during eval: ~2 GB
Safety margin: ~8 GB

Typical times (an estimate based on community benchmarks; measure in the lab before promising anything):

Dataset	Technique	Adapter rank	Approximate time
1,000 SFT examples	LoRA r=16	16	20-40 min
5,000 SFT examples	LoRA r=32	32	2-4 h
2,000 DPO pairs	LoRA r=16	16	1-2 h
5,000 KTO examples	LoRA r=32	32	3-5 h

This puts the iteration cycle, a change in the dataset, retrain, eval, look at the number, in the range of a working day. Enough to validate a hypothesis before moving anything to the production cluster.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink) for production

With a cluster of this order the whole picture changes. You can:

Train LoRA on a 70B in BF16 with no quantisation with tensor parallel = 4.
Do full DPO with a resident reference model when the reference is quantised to FP8, or move to SimPO / ORPO, which remove that intermediate model and simplify VRAM planning (see the table of techniques above).
Support multi-tenant fine-tuning: several customer adapters training in parallel in separate logical pipelines, each isolated in a different Postgres partition with its own ACLs.
Serve multi-LoRA with --max-loras 8 on top of the base model without concurrency lowering throughput perceptibly.

The practical budgeting rule: over a 12-month horizon, a team with this cluster can run ~150-200 continuous fine-tuning cycles (training + eval + canary + promotion or discard) if the discipline around the dataset and the eval gates is strict. If it is not, it will run twice as many but with half the usefulness.

Position within the architecture: what this article covers and what it does not

To place the scope: the cycle drawn at the start has seven boxes, all of them covered here in their mechanics. Three cross-cutting layers are deliberately left out, and they are the ones that end up separating a pipeline that works technically from one that survives an audit:

Cryptographic provenance and traceability. We have mentioned dataset_snapshot and pg_audit, but the complete mechanics, the set_hash over the examples, the integration with the EU AI Act, the frozen query_sql as proof of what trained the model, is enough for a whole analysis.
Judge calibration. We have assumed LLM-as-judge works. It needs calibrating against a human rubric on at least 100 cases per critical suite before trusting it. Without that calibration, the eval gates are theatre.
The forgetting problem. What happens if a user exercises their GDPR right to be forgotten and their interactions were part of the dataset of an adapter already in production? There is no clean solution. There are options, incremental retrain, sample-level machine unlearning, negative LoRA, and it is worth knowing them before a customer asks.

What we have not covered (upcoming articles)

Cryptographic provenance on Postgres: how set_hash and a frozen query_sql compose an auditable chain of custody under the EU AI Act.
Honest judge calibration: why score > 0.85 means nothing without a human baseline, and how to build that baseline without it costing a month of work.
The forgetting problem in adapters: sample-level machine unlearning, incremental retrain and other techniques for answering GDPR without throwing the adapter away.
Online DPO and on-policy continual learning: the state of 2026 research (Fast-Slow Chasing, RLOO, iterative on-policy) and why it is not production yet.

References

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022).
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023).
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
Hong et al., ORPO: Monolithic Preference Optimization without Reference Model (2024).
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM.
Official vLLM Multi-LoRA documentation: https://docs.vllm.ai/en/stable/features/lora/.
Official pgvector 0.8 documentation: https://github.com/pgvector/pgvector.
TRL (HuggingFace) docs: https://huggingface.co/docs/trl.
EU AI Act, consolidated text and application timeline: https://artificialintelligenceact.eu/.

MLOps specific to LLMs in 2026: the landscape of three modalities, six stages and ten tools that make them work

Thu, 21 May 2026 05:30:00 +0200

TL;DR

This is the blog’s fourth series and it is called MLOps specific to LLMs. It takes the operational craft of traditional MLOps (reproducible pipelines, model registries, dataset versioning, eval gates, controlled deployments) and redraws it for a world where the model is probabilistic, the outputs are subjective, the dependencies include external vendors that update weights without warning, and the “application” is not a model but an orchestration of models, embeddings, retrievers, guardrails and routers. Gartner predicts that more than 50% of enterprise GenAI deployments will fail before the end of 2026, and the main cause is not the model: it is that deterministic software assumptions were applied to probabilistic systems. This post opens the series with the frame: the seven structural differences between LLMOps and classic MLOps; the six-stage pipeline (data → tune → eval → deploy → observe → retrain); the three modalities of preparing a model (continuous fine-tuning, RAG over datalakes, agent training) with their decision matrix, where 60% of 2025-2026 deployments use hybrid because each modality solves a different problem: “fine-tune for behaviour, RAG for volatile knowledge”; and the 2026 tool landscape, which already forms reasonably stable layers: MLflow 3.10 (March 2026) as a GenAI-aware registry, W&B Weave and ZenML for tracing and pipelines, Kubeflow + KServe vLLM 0.8.1+ for serving, BentoML for flexibility, DVC + lakeFS (joined since November 2025) for data, Langfuse for prompts and observability. The three following posts will go down into the detail of the most critical pieces.

This is the opening of series 4: MLOps for LLMs. It continues the tradition of the previous series: LLM inference (the first), eBPF (the second) and post-tracing (the third). Here we enter the discipline that ties all the pieces together: how an LLM system is operated in production over months, not just deployed once.

The analogy: the SRE’s craft redrawn

Anyone who has worked for years as an SRE or a platform engineer recognises the classic pillars: reproducibility (same code + same data + same config = same result), observability (what happens can be measured), safe rollback (if something goes wrong, I go back within minutes), gradual rollout (the new thing goes to 1% before it goes to 100%). These pillars are not negotiable. The question is whether they hold up when the central component is an LLM.

The answer is: same pillars, radically different mechanics. Reproducibility: versioning code and data is no longer enough; you have to version prompts, retrieval configurations, snapshots of the vendor’s model (which change without warning). Observability: error and latency metrics are no longer enough; you have to measure subjective quality via LLM-as-judge and embedding drift. Rollback: dropping the binary’s version is no longer enough; you have to keep the old model cached because loading a new one takes minutes. Gradual rollout: a % of traffic is no longer enough; you have to decide what % of what kind of queries by segment.

It is the same craft, practised with partly new tools and reflexes. MLOps specific to LLMs, or “LLMOps”, as the field has christened itself, is the discipline that codifies those reflexes.

The seven structural differences between LLMOps and traditional MLOps

Before going down to the pipeline, let us fix the differences that make this territory new rather than a mere continuation. Each one has concrete practical consequences.

1. Non-deterministic outputs

Traditional MLOps: the model receives structured input and returns a bounded, reproducible prediction. Same input → same output. Unit tests work.

LLMOps: same input → different output every time (through sampling, through temperature, through the order of tools invoked, through the retrieval context that changed). The idea of a “unit test” breaks.

Operational consequence: tests over properties (was the tone maintained?, does it mention the source?, does it respect the JSON schema?), not over equality. Statistical evals over a distribution, not over samples.

2. Behaviour metrics, not statistical accuracy

Traditional MLOps: F1, accuracy, AUC, RMSE. Metrics with a clear number.

LLMOps: subjective rubric scores (G-Eval, faithfulness, helpfulness, toxicity), judge LLMs, human feedback. The “number” depends on who judges.

Operational consequence: platforms have to treat evals as versioned artifacts, not just “model v3 scored 0.87”, but “model v3 evaluated with the judge claude-3-5-sonnet-20251022 over the dataset gold-rag-v7 with the prompt judge-v2 scored 0.87”. Versioning the judge is as important as versioning the model being evaluated.

3. The model is an external dependency, not an internal asset

Traditional MLOps: you train the model, it lives in your registry, it does not change until you change it.

LLMOps: the base model belongs to Anthropic, OpenAI, Google, Meta. They change it on you without warning. The claude-3-5-sonnet version that answered well yesterday answers somewhat differently today.

Operational consequence: drift detection becomes far more critical (previous post). Pinning to specific snapshots (claude-3-5-sonnet-20251022) when the vendor allows it. For high-commitment apps, self-hosting the base model to guarantee reproducibility.

4. The application is an orchestration, not a model

Traditional MLOps: an app calls a model and consumes its output.

LLMOps 2026: an app connects foundation model + LoRA adapters + retrievers + vector stores + guardrails + routers + tool servers (MCP) + evaluators, all composing the final behaviour. Any component can degrade the result.

Operational consequence: cross-component debugging requires distributed tracing with OTel (covered in previous posts). The registry does not only store “the model” but the composition: which prompt version + which adapter + which vector store + which retriever config.

5. Cost per inference, not per training

Traditional MLOps: the high cost is training; serving is cheap. You optimise training.

LLMOps: the high cost is serving (every token costs, every call to the vendor is paid for, the GPUs doing the serving are switched on 24/7). You optimise inference.

Operational consequence: cost accountability per tenant, per agent, per tool. Metrics such as gen_ai.usage.input_tokens aggregated at customer and product level. Model decisions based on cost per query, not only on quality.

6. GPU-heavy infra with specific primitives

Traditional MLOps: CPU + some GPU for training. Standard Kubernetes.

LLMOps: Hopper/Blackwell SXM GPUs, NVLink/NVSwitch, tensor parallel, paged attention, KV cache. Infra that only fits into Kubernetes with primitives such as LeaderWorkerSet, GPU Operator, KEDA with LLM metrics (covered in vLLM on Kubernetes).

Operational consequence: the orchestration stack includes specialised operators (OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, see LLM K8s Operators) that traditional MLOps does not contemplate.

7. RLHF and human feedback as a first-class citizen

Traditional MLOps: human feedback means labelling data before training.

LLMOps: human feedback lives inside the model in production, whether through RLHF of the foundation models (Anthropic, OpenAI), through RLAIF, through DPO, or through explicit user feedback that is fed back into fine-tuning.

Operational consequence: bidirectional production → training pipelines. Datasets grow with real incidents. Model decisions are taken with continuous feedback, not in a training project every N months.

Why Gartner predicts 50%+ failures

Gartner published that more than 50% of enterprise GenAI deployments will fail before 2026. The causes are not technical ones about the model but about the system:

Hallucinated outputs from poor grounding: badly designed RAG, poor retrieval, insufficient context.
Data architectures that are not ready: companies have data in silos, with no stable schemas, with no controlled freshness. Connecting an LLM to this data without a serious pipeline produces erratic answers.
Lack of structured workflows for prompt-driven systems: teams that treat prompts as code in hardcoded strings, with no versioning, no tests, no gates.

The conclusion the field draws: LLMOps is not optional. Companies that deploy GenAI without operational discipline fall into one of the three failure modes. Those that apply it (MLflow/W&B for tracking, DVC/lakeFS for data, Langfuse for prompts and evals, KServe or vLLM Production Stack for serving, drift detection in production) are the ones that keep the system working six months after the first release.

The six-stage LLMOps pipeline

On to the pipeline. The six stages that any serious LLM system goes through, in order:

[1. Data] → [2. Tune] → [3. Eval] → [4. Deploy] → [5. Observe] → [6. Retrain]
│
└─→ back to 1

Each stage is an operational domain of its own, with its own tools and traps:

Stage 1 — Data

What happens: ingestion, cleaning, curation, versioning, indexing of the corpus. It is where real projects suffer most, because companies have data in heterogeneous silos.

Typical sub-tasks: extraction from source (CDC over Kafka, batch from data lakes, scraping), cleaning (PII removal, dedup, format), curation (labelling for fine-tuning, golden datasets for eval), versioning (DVC + lakeFS), indexing (embeddings + vector store for RAG).

Traps: schema drift at the source, undetected PII, poor dedup that injects redundancy into training, a vector store that is not updated.

Stage 2 — Tune

What happens: preparing the model for your use case. Three modalities (we go deeper into them shortly): fine-tuning, RAG, agent training.

Typical sub-tasks: base model selection, adapter preparation (LoRA, QLoRA), training loop with continuous eval, hyperparameter sweep (Optuna, W&B Sweeps), saving the checkpoint.

Traps: catastrophic forgetting if the fine-tuning is very aggressive, overfitting to the golden dataset, no independent validation set.

Stage 3 — Eval

What happens: validating that the model + adapters + RAG configuration is acceptable before promoting. Covered in Evals.

Typical sub-tasks: running an eval framework (DeepEval, Promptfoo, Ragas) against the golden dataset, judge LLM evaluations, human review over a sample, gates with thresholds.

Traps: a golden dataset that ages, an uncalibrated judge, evals that pass in CI but fail in production because of distribution shift.

Stage 4 — Deploy

What happens: going from “the model evaluated well” to “the model serves real traffic”. Covered in LLM K8s Operators.

Typical sub-tasks: serving with vLLM/SGLang/TRT-LLM, runtime configuration, gradual rollout (canary, shadow, blue-green), routing between models (LiteLLM, OpenRouter, LangChain routers).

Traps: a naive rolling update that cuts sessions, autoscaling by CPU% that does not respond to LLM metrics (covered), a new model that performs worse in production than in eval.

Stage 5 — Observe

What happens: seeing what is happening in real time. Covered in the whole post-tracing series and the eBPF series.

Typical sub-tasks: tracing (Langfuse, LangSmith, Phoenix, OpenLLMetry), metrics (TTFT, TPOT, queue depth, cost per query), active guardrails (NeMo, Llama Guard), drift detection (Evidently, NannyML, WhyLabs).

Traps: cardinality explosion in metrics, batch evals with no tail-sampling over real traces, drift that is ignored until an incident makes it real.

Stage 6 — Retrain

What happens: closing the loop. Production feedback (incidents, worst-scoring cases, detected drift) generates new data to go back to stage 1.

Typical sub-tasks: extraction of problematic logs, human labelling of the sample, incorporation into the golden dataset, re-fine-tuning if applicable, decision on a new release.

Traps: an “open” loop where production never informs the dataset, human feedback that gets lost, lack of a clear retrain cadence.

The three modalities of “preparing the model”

Stage 2 (Tune) is where there is most confusion. In 2026 three modalities coexist, each solving a different problem:

Fine-tuning

What it does: modifying the model’s weights (or those of a LoRA/QLoRA adapter on top) so that it learns specific behaviour patterns: tone, output structure, idiomatic decisions of the domain.

When: when your main failure is behavioural inconsistency between calls. The model sometimes answers formally, sometimes not; sometimes it structures the JSON, sometimes not; sometimes it follows the company’s conventions, sometimes it invents them. Fine-tuning stabilises it.

When NOT: when what you need is up-to-date knowledge. Fine-tuning fixes knowledge in frozen weights; the day after the fine-tuning, the model knows nothing new.

RAG (Retrieval-Augmented Generation)

What it does: leaving the model intact and, on each call, retrieving fresh context from a knowledge base (vector store + lexical search, typically) and passing it to the model so that it answers on that basis.

When: when the knowledge you need is dynamic or very large. Documentation that changes, a product catalogue that is updated, an internal knowledge base that grows.

When NOT: when the problem is behavioural (RAG does not teach the model how to behave, it only gives it information). Or when retrieval is so noisy that the context arriving is worse than nothing.

Agent training

What it does: going beyond conventional fine-tuning with Reinforcement Learning techniques. RFT (OpenAI’s Reinforcement Fine-Tuning), classic RLHF, RLAIF (with AI feedback), DPO (Direct Preference Optimization) over datasets of (good, bad) pairs.

When: when the model needs to learn complex multistep trajectories: when to choose each tool, how to decompose a task, when to ask the user for confirmation. It is what is turning Claude, Gemini and GPT into agents capable of long tasks.

When NOT: when your case is simple chat or RAG. It is overkill, expensive and complicated for problems that the previous modalities solve.

Decision matrix

Observed problem	Modality
Inconsistent answers in tone/structure	Fine-tuning
The model makes things up (hallucinates)	RAG
Out-of-date knowledge (>1 year)	RAG
The model chooses tools badly	Agent training (RLAIF/RFT)
Mixed behaviour + knowledge	Hybrid (fine-tune + RAG)
Multi-step trajectory fails	Agent training
A specific language/regional style	Fine-tuning

The 2026 verdict: hybrid is the default

Multiple reports agree that in 2025-2026, around 60% of production projects use hybrid: fine-tuning for behaviour + RAG for knowledge. The key insight:

Fine-tune for behaviour (brand voice, decision protocol, output structure); use RAG for volatile knowledge you need the model to cite. Do not force a single tool to do both jobs.

A practical observation: the biggest quality improvements of 2025-2026 come from better reranking in RAG (cross-encoders), not from better embeddings. Rerankers add 15-35% of quality with little complexity.

On cost: combined fine-tuning + RAG tends to be 30-50% cheaper than pure RAG with frontier models at high volume, because the fine-tuned model can be smaller and cheaper while keeping equivalent quality.

The 2026 tool landscape

On to the concrete pieces, grouped by function. The field has matured enough for each piece to have 2-3 reasonable options and a couple of leaders.

Experiment tracking and model registry

MLflow remains the de facto standard, now with LLM-specific traction. MLflow 3 was published in June 2025; version 3.10.1 (March 2026) added:

A GenAI Overview dashboard with ready-made metrics for LLM apps.
Multi-workspace support for large teams.
Cost tracking in traces (gen_ai.usage.* aggregated per experiment).
MemAlign: a new specific eval algorithm.
Integrated native OpenTelemetry tracing.
First-class support for LangChain, LlamaIndex, AutoGen as frameworks.

MLflow treats prompts and agents as first-class citizens alongside classic models. It is the biggest change relative to MLflow 2.x.

Weights & Biases (W&B) with its LLM-specific product Weave offers tracing + eval + debug with a very polished UI. More commercial, less self-host friendly, but excellent UX.

ZenML is the piece that most cleanly integrates “classic MLOps + emerging LLMOps” into a single framework. Its automatic artifact versioning captures prompt templates, retrieval chunks and agent conversation histories with no extra work. Open-source. The most complete unification option that exists.

Dataset versioning

DVC remains the OSS standard. It extends Git to large files and pipelines. Important news from November 2025: lakeFS acquired DVC, consolidating the two OSS data versioning projects under one organisation. The combined roadmap is oriented towards LLM training and RAG datalakes specifically.

Typical pattern: Git for code + DVC for data/models + MLflow or W&B for experiment tracking + registry. Few teams use just one; the combination is what covers the cycle. Detailed in the dedicated post on data versioning for LLMOps — the four artifacts to version separately, schema contracts, end-to-end lineage, and a golden eval set with a strict holdout.

Prompt versioning and observability

Covered in depth in the AgentSight post where we go deeper into Langfuse as the OSS reference. Summary here:

Langfuse: MIT, self-host, prompt management with v1/v2/v3 versioning + labels + cache + linkage with traces.
LangSmith: if your stack is LangChain.
Arize Phoenix: ELv2, OTel-native.

Pipeline orchestration

For the LLMOps pipeline steps, the dominant options:

Kubeflow Pipelines: the K8s-native standard. KServe (the serving part of Kubeflow) has its vLLM runtime upgraded to v0.8.1+ with support for reasoning models, tool calling, embeddings, reranking, Llama 4 and Qwen 3.
ZenML: already mentioned; also a pipeline orchestrator.
Metaflow (Netflix-originated): Python-first pipelines, less LLM-specific but workable.
Argo Workflows: a pure K8s OSS alternative.
Flyte: Kubernetes-native, OSS.

Serving

Covered in depth in vLLM on Kubernetes and LLM K8s Operators. Summary:

vLLM Production Stack: a curated Helm chart.
KServe vLLM runtime: K8s-native, vLLM 0.8.1+ with full agentic support.
BentoML: flexible serving, popular in startups for its simplicity.
NVIDIA Dynamo: the successor to Triton.
llm-d: CNCF Sandbox.
OME: an LMSYS operator with native SGLang.

Evals and guardrails

Covered in Evals and Guardrails. Ultra-short summary:

Evals CI: DeepEval, Promptfoo, Ragas.
Evals platform: Langfuse, LangSmith, Phoenix, Braintrust.
Guardrails: NeMo Guardrails, Llama Guard 4, Llama Prompt Guard 2, LLM Guard, Lakera.

Drift detection and observability

Covered in the closing eBPF post. Summary:

Drift: Evidently AI, NannyML, WhyLabs.
Tracing: Langfuse, OpenLLMetry, Phoenix.
eBPF: AgentSight, Hubble, Tetragon, ProfInfer.

The typical 2026 stack table

Stage	Dominant piece	Alternatives
Data ingestion + versioning	DVC + lakeFS (unified Nov 2025)	Pachyderm, Quilt
Vector store / RAG index	Milvus, Qdrant, pgvector, Weaviate	LanceDB, Pinecone, Chroma
Experiment tracking	MLflow 3.10	W&B Weave, Neptune
Pipeline orchestration	Kubeflow + Argo	ZenML, Metaflow, Flyte
Model registry	MLflow registry	W&B Models, KServe ModelMesh
Prompt versioning	Langfuse	LangSmith, MLflow Prompts
Serving	vLLM Production Stack	KServe, BentoML, Dynamo, llm-d, OME
Evals CI	DeepEval, Ragas	Promptfoo, OpenAI Evals
Evals platform	Langfuse, Phoenix	LangSmith, Braintrust
Guardrails	NeMo + Llama Guard	LLM Guard, Lakera
Tracing	OpenLLMetry + Langfuse	Phoenix, LangSmith
Drift detection	Evidently AI	NannyML, WhyLabs
eBPF observability	AgentSight + Tetragon + Hubble	(new territory, few alternatives)

13 pieces. No org uses them all; any serious org uses at least six. This is the current LLMOps stack.

The operational reality: nobody uses a single tool

Multiple comparisons agree on one thing: the teams that win combine. Recurring patterns:

ZenML to orchestrate + MLflow for tracking + KServe for serving: the most popular OSS stack in companies coming from classic MLOps.
Kubeflow + W&B + BentoML: for teams with a research focus.
Langfuse + DeepEval + Phoenix + LiteLLM: for pure-LLM teams with no classic MLOps background.
MLflow + DVC + Argo + KServe: an idiomatic cloud-native stack without additional LLM-specifics (with its limitations).

The choice depends on the team’s background, on the licensing model they can afford, on the level of self-hosting they need, and on which frictions blocked them most in previous projects. There is no “one right answer”; there is a stable meta-pattern of layers worth covering.

Common operational traps

Treating the prompt as text in code

Hardcoding prompts in strings in the repo. Changing them requires a PR + redeploy. Result: teams that do not iterate on prompts because every change costs hours of pipeline. Solution: externalised prompt management (Langfuse, MLflow Prompts) with versioning, labels, hot-reload.

Skipping dataset versioning

“DVC is complicated, we will add it later”. Result: two months later, nobody knows which dataset trained which model. Impossible to reproduce incidents. Solution: DVC + lakeFS from day 1, even if only with a small subset.

Mixing layers in the same pipeline

Teams that put ingestion, fine-tuning, eval and deploy into a single giant pipeline. When something fails, the whole pipeline fails. Solution: independent pipelines per stage, with versioned artifacts as the interfaces between them.

Tracking with no structure

Logging everything to stdout and “we will look at it in CloudWatch later”. Result: impossible to correlate, compare, debug. Solution: OTel from day 1 with gen_ai.* semantic conventions.

Evals that block nothing

You have evals, you run them, you look at them, but they do not prevent the deploy if they drop. Eventually it drops gradually and nobody notices. Solution: eval gates in CI/CD that block the merge if critical metrics regress by more than X%.

No retrain cadence

You launch v1 and never go back to the model. Six months later, drift has degraded it but the team is on other projects. Solution: a formal retrain cadence (monthly, quarterly) tied to the production incident queue.

Invisible vendor lock-in

You start with the OpenAI API + LangSmith + Pinecone. When you want to self-host, you discover that migrating is a 3-month project. Solution: abstraction layers (LiteLLM, OpenLLMetry) and vendor-neutrality from the start.

What is coming in the following posts of the series

Post 2 — RAG over datalakes with Kafka: end-to-end technical architecture — the most hands-on one. Kafka as source-of-truth, Flink CDC, embedding pipelines, continuous indexing in Milvus/Qdrant, a complete example with real numbers and manifests.
Post 3 — The six-stage LLMOps pipeline: global architecture — the master map of the complete system with a reusable “you are here” SVG for the following posts. A deep dive into each of the six stages (Data, Tune, Eval, Deploy, Observe, Retrain).
Post 4 — PostgreSQL + Qdrant in the ingestion stage — synchronisation patterns (dual-write, outbox + CDC, event-driven), a complete microservices architecture, a Qdrant cluster manifest.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / alignment runtime, continuous fine-tuning in depth, edge LLMs.
Synthesis post — Anatomy of an LLM request in production, May 2026 — the complete journey of a real request through the six stages and the two cross-cutting components, with cross-links to each dedicated post. It works as a mental map of the blog and as an integrator’s guide.
Parallel OSS vs hyperscalers catalogue — The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure — for each stage of the pipeline, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, identification of real gaps and the multi-tenant chatbot ported to an AWS stack component by component. A neutral editorial stance.
OSS catalogue entry by entry — The OSS catalogue for LLMOps in six stages: entry by entry — the zoom in on the open source side: ~150 words of description per core tool (vLLM, Langfuse, DVC, Qdrant, Airflow, NeMo Guardrails, Presidio…), licence and governance, a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.

References

LLMOps vs MLOps:

Tools:

MLflow — registry + tracking + serving.
Weights & Biases Weave — LLM tracing.
ZenML — pipeline orchestration MLOps + LLMOps.
Kubeflow — K8s-native MLOps.
KServe — model serving K8s.
BentoML — flexible serving.
Metaflow — Netflix’s pipelines.
DVC — dataset versioning.
lakeFS — enterprise data versioning, acquired DVC in Nov 2025.

2026 comparisons:

RAG vs Fine-Tuning:

Cross-references (the three previous series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.

Fine-Tuning on lo0 — Tech Blog

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

TL;DR

The end-to-end flow

Phase 0 — Environment and versions

Phase 1 — Preparing the dataset

Phase 2 — The training script

BitsAndBytesConfig — how the base is quantised

LoraConfig — the shape of the adapter

SFTConfig — the memory budget and the loop

Phase 3 — Launch and monitor

Phase 4 — Validating the adapter

Phase 5 — Versioning the adapter as an artefact

Phase 6 — Serving in multi-LoRA with vLLM

Serving multi-adapter vs merging per task

Checklist of operational gotchas

Applied to on-premise infrastructure

See also

References

QLoRA and multi-LoRA at the limit on small models

TL;DR

The analogy: the frozen guitar and the swappable pedalboard

The bare mechanism: LoRA, and why you can train on a 4-bit base

NF4: why a new format instead of INT4

Double quantisation and paged optimizers

“Aggressive” training: very low rank and QA-LoRA

The maths that matters

The fit with small models and sovereignty

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

You are here: DEPLOY

TL;DR

The analogy: the single translator with a thousand glossaries

The bare mechanism: what a LoRA does and why it can be served multi-tenant

SGMV: the kernel that holds it all up

The maths that matters

The real implementations in May 2026

Combined pattern with quantisation and disaggregated serving

Operational pitfalls

Implications on on-premise hardware

Typical production stack

What we have not covered

See also

References

Modern alignment: DPO, KTO, ORPO and SimPO — the sommelier who learns without a reward model

TL;DR

You are here: TUNE

The analogy: the sommelier who trains the palate without a theory book

Why DPO exists: Rafailov’s trick

DPO with real numbers

KTO: when you only have 👍/👎

ORPO: SFT and preferences in a single pass

SimPO: do you really need a reference model?

Decision table: which one to use and when

The three biases that break the method

Length bias

Position bias (in dataset curation)

Distribution shift between the data and π_ref

Implications on on-premise hardware

On an RTX 4090 (24 GB)

On a generic 4×H100 SXM cluster (320 GB, NVLink)

What we have not covered (upcoming articles)

See also

References

Continuous fine-tuning in production: from real traffic to the deployed adapter

TL;DR

You are here: Tune + Retrain

The analogy: the restaurant that tunes its menu

The cycle, taken apart

Why continuous fine-tuning (and why it is not RAG)

The four techniques according to the signal you capture

Postgres as the nervous system of the pipeline

A concrete schema

The adapter registry

Generating DPO and KTO datasets from real traffic

`BitsAndBytesConfig` — how the base is quantised

`LoraConfig` — the shape of the adapter

`SFTConfig` — the memory budget and the loop

Distribution shift between the data and `π_ref`