QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

This is the operational companion to QLoRA and multi-LoRA at the limit on small models. That post takes apart the why, NF4, double quantisation, paged optimizers, the adapter maths; this is the how, with commands you copy and paste. If you have not read the fundamentals one, read it first: here we assume you know what an adapter is, why the base lives in 4-bit and why the gradient only touches the adapter.

TL;DR

A reproducible procedure in five phases: (1) pin the environment with fixed versions; (2) prepare the dataset in chat format; (3) train the QLoRA adapter with TRL + PEFT on an RTX 4090 (24 GB, Ada Lovelace) using gradient checkpointing, gradient accumulation and paged_adamw_8bit; (4) validate and version the adapter as an artefact of megabytes; (5) serve it in vLLM with --enable-lora, loading it hot without restarting the server and resolving it from object storage. All on-premise, on consumer hardware, without a single data item leaving the perimeter. What follows are the exact commands and the memory budget that separates “it fits” from “OOM”.

The end-to-end flow

Phase 0 — Environment and versions

QLoRA is sensitive to the versions of bitsandbytes, transformers, peft and trl: misaligned combinations give dequant errors or adapters that do not load in vLLM. Pin the environment and do not touch it mid-campaign. Reference versions as of June 2026 (check the specific ones in your index; the exact pin matters less than the coherence between them):

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Training (producer)
pip install "torch>=2.4" \
            "transformers>=4.50" \
            "peft>=0.14" \
            "trl>=0.15" \
            "bitsandbytes>=0.45" \
            "accelerate>=1.2" \
            "datasets>=3.2"

# Serving (consumer) — in its own environment/image
pip install "vllm>=0.8"

What each piece does and why it is pinned:

The golden rule: coherence between the four above (transformers, peft, trl, bitsandbytes) matters more than the exact number of each. Pin them when starting a campaign and do not move them until it closes.

Check that the GPU and CUDA are healthy before starting; a badly compiled bitsandbytes shows up late:

python -c "import torch, bitsandbytes; print(torch.cuda.get_device_name(0), torch.cuda.is_available())"
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

For 100 % sovereignty: download the base once from your internal Hugging Face mirror (or a MinIO with the weights) and export HF_HOME to a local volume. Nothing in this flow needs to leave the perimeter.

Phase 1 — Preparing the dataset

The canonical format for a conversational task is JSONL, one conversation per line, with the model’s chat template. Do not invent your own format: use the base tokenizer’s chat_template, because any mismatch between how you train and how you serve degrades quality silently.

{"messages":[{"role":"system","content":"You are a network support assistant."},{"role":"user","content":"The north wing AP does not come up after the outage."},{"role":"assistant","content":"First confirm the PoE on the port..."}]}
{"messages":[{"role":"user","content":"Generate the VLAN change for customer 42."},{"role":"assistant","content":"interface GigabitEthernet0/3\n switchport access vlan 42..."}]}

What each field is and why:

Operational rules that save grief: watch the ratio of examples (a well-curated narrow-task dataset of 2,000-20,000 examples performs better than 200,000 noisy ones), deduplicate, and set aside 5-10 % as a validation split that does NOT go into training. Building the corpus from production signal is covered by Retrain: closing the loop.

Phase 2 — The training script

A minimal, complete script with TRL + PEFT. It trains an r=8 adapter on an 8B SLM quantised to NF4. Each block has its rationale commented.

# train_qlora.py
import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen3-8B"           # or whichever SLM you serve; ALWAYS use the same one in train and serve
OUT  = "adapters/soporte-redes-v1"

# 1) Base frozen and quantised to 4-bit NF4 with double quantisation
bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # NormalFloat, quantile-optimal for Gaussian weights
    bnb_4bit_use_double_quant=True,       # quantises the scale constants -> ~0.37 bits/param less
    bnb_4bit_compute_dtype=torch.bfloat16 # the matmuls run in BF16 after dequant on the fly
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map={"": 0}
)

# 2) The adapter: low rank, attention projections only (aggressive). Raise target_modules if the eval asks for it.
peft_cfg = LoraConfig(
    r=8, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

ds = load_dataset("json", data_files={"train": "data/train.jsonl",
                                      "eval":  "data/eval.jsonl"})

# 3) Training config designed to fit in 24 GB
cfg = SFTConfig(
    output_dir=OUT,
    per_device_train_batch_size=1,        # small real batch
    gradient_accumulation_steps=16,       # EFFECTIVE batch = 1*16 = 16, without paying its VRAM all at once
    gradient_checkpointing=True,          # recomputes activations in backward: trades compute for memory
    optim="paged_adamw_8bit",             # paged optimizer: the airbag against VRAM peaks
    learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
    num_train_epochs=3, bf16=True,
    max_length=2048,                      # bounds the sequence: activations scale with it
    logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=200,
    report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
                     train_dataset=ds["train"], eval_dataset=ds["eval"],
                     processing_class=tok)
trainer.train()
trainer.save_model(OUT)                   # saves ONLY the adapter (MB), not the base

BitsAndBytesConfig — how the base is quantised

LoraConfig — the shape of the adapter

SFTConfig — the memory budget and the loop

The four pieces that make it fit on a 4090 are: per_device_train_batch_size=1 + gradient_accumulation_steps (a large effective batch without its memory cost all at once), gradient_checkpointing=True (recomputing activations instead of storing them) and optim="paged_adamw_8bit" (paging states to RAM at the peaks). Remove any of the three with long sequences and you will see the OOM.

A declarative alternative with Axolotl if you prefer YAML over Python (same result):

base_model: Qwen/Qwen3-8B
load_in_4bit: true
adapter: qlora
lora_r: 8
lora_alpha: 16
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
sequence_len: 2048
micro_batch_size: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
optimizer: paged_adamw_8bit
learning_rate: 0.0002
num_epochs: 3
bf16: true
datasets:
  - path: data/train.jsonl
    type: chat_template

Phase 3 — Launch and monitor

# Simple launch on one GPU
python train_qlora.py

# In another terminal: watch VRAM. If it gets close to the ceiling, lower max_length or raise grad accumulation.
watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Approximate VRAM budget when training the 8B on the 4090, and what to touch when it gets tight:

Table of quick OOM remedies, in order of cost: (1) lower max_length; (2) confirm gradient_checkpointing=True; (3) raise gradient_accumulation_steps and lower per_device_train_batch_size to 1; (4) use paged_adamw_8bit (already in the script); (5) as a last resort lower r. If after all that it still does not fit, the sequence or the model is too large for 24 GB: either you bound it, or you move up the hardware.

Phase 4 — Validating the adapter

Never promote an adapter on the strength of the training loss. Measure against the validation split you set aside and against a handful of real prompts.

# quick_eval.py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, "adapters/soporte-redes-v1")  # base + adapter

msgs = [{"role": "user", "content": "The north wing AP does not come up after the outage."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(0)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0], skip_special_tokens=True))

For a serious verdict, run the adapter through your eval suite (the layer described by LLM evals) and compare against the base without an adapter and against the previous version of the adapter. Promote only if it wins on the task metric without regressing on safety/format.

Phase 5 — Versioning the adapter as an artefact

The adapter is a pair of files of a few MB (adapter_model.safetensors + adapter_config.json). Treat it as a versioned, signed, traceable artefact, not as a loose file.

# Reproducible checksum + upload to internal object storage (MinIO/S3)
sha256sum adapters/soporte-redes-v1/adapter_model.safetensors > adapters/soporte-redes-v1/SHA256
aws --endpoint-url https://minio.interno s3 cp \
    adapters/soporte-redes-v1/ s3://adapters/soporte-redes/v1/ --recursive

A convention that works: s3://adapters/<task-or-customer>/<version>/. Immutable per version, with its SHA256. Deleting a customer is deleting a prefix of a few MB, not retraining anything. Versioning 500 adapters costs what versioning 500 heavy configuration files costs.

Phase 6 — Serving in multi-LoRA with vLLM

The consumer loads one shared base and applies the adapter delta per request. Startup with static adapters declared:

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3-8B \
  --enable-lora \
  --max-loras 8 \              # max nº of DIFFERENT adapters per batch (not the total loadable)
  --max-lora-rank 8 \          # = the maximum rank of your adapters; do not inflate it (wastes memory)
  --max-cpu-loras 64 \         # adapters cached in RAM for fast swap to VRAM
  --lora-modules soporte-redes=/srv/adapters/soporte-redes/v1

Each flag, what it controls and how to size it:

--max-loras limits the different adapters per batch, not how many you can have loaded; the bulk lives on CPU (--max-cpu-loras) and is paged to VRAM on demand. Set --max-lora-rank to the real rank (8 here): inflating it wastes memory and performance. Requests pick the adapter through the model field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "soporte-redes",
  "messages": [{"role":"user","content":"The north wing AP does not come up after the outage."}]
}'
# model:"Qwen/Qwen3-8B" (no adapter) uses the bare base on the same server

Hot loading of a new adapter without restarting (thanks to VLLM_ALLOW_RUNTIME_LORA_UPDATING=True):

curl -X POST http://localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" -d '{
  "lora_name": "cliente-42",
  "lora_path": "/srv/adapters/cliente-42/v3"
}'
# and to free VRAM/CPU when a customer goes idle:
curl -X POST http://localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" -d '{"lora_name":"cliente-42"}'

For multi-tenant at scale, avoid declaring hundreds of adapters by hand: the LoRAResolver resolves and loads the adapter from local storage or S3 the first time an unknown model arrives, so the server stays lean and the adapters are pulled lazily from your MinIO. The internals of how thousands of concurrent adapters are batched (SGMV kernels, unified paging, the heterogeneous gather/scatter) are in Multi-LoRA serving; this runbook only switches them on. To squeeze the base’s decode throughput on a 4090, combine this with Optimising decode in vLLM.

Serving multi-adapter vs merging per task

Two deployment architectures, and the procedure changes:

Serving multi-LoRA (the above). A shared base + N hot adapters. It is the default sovereign pattern: minimum footprint, per-customer isolation, hot-swap. Use standard QLoRA and merge nothing.

Merging per task. If you want a single quantised-and-adapted artefact per task (no adapter at runtime), do not merge a standard QLoRA adapter into the 4-bit base: the merge reintroduces precision that NF4 does not represent and on requantising you lose part of what was learnt. For that case, train with QA-LoRA (quantization-aware), which merges cleanly onto a quantised base. It is an architecture decision, not a quality one; the conceptual detail is in the fundamentals post.

Checklist of operational gotchas

• Coherent chat template between training and serving. The most common and most silent mismatch: you train with one chat_template and serve with another. Use the base’s on both sides.
• Exactly the same base (revision included) in train and serve. An adapter trained on Qwen3-8B is not valid on another revision of the model.
• --max-lora-rank ≥ the rank of ALL the adapters served together, but no more: inflating it wastes VRAM.
• KV budget vs --max-loras. The bottleneck in serving is not the adapters (MB), it is the KV cache and concurrency; see Inverted roofline for the SLM regime.
• r too low = underfitting if the task demands rewriting a lot of behaviour. Raise r only if the eval asks for it.
• Do not promote on training loss. Validate against the reserved split + real prompts + a safety regression.
• Version and make immutable every adapter with its SHA256; never overwrite a served version.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB) the same machine is producer and consumer: you train a customer’s adapter in hours and serve it on the same server on top of the shared base. It is the canonical case for multi-tenant demos and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8) QLoRA stops being necessary in order to fit, but it serves to parallelise production (several adapter jobs at once) and to keep the quantised format consistent between training and serious serving of hundreds of concurrent adapters. The base can run in native FP8; the mechanics of the runbook do not change, only the scale.

See also

• QLoRA and multi-LoRA at the limit on small models — the fundamentals post: the why of NF4, double quantisation, paged optimizers and the adapter maths. This runbook is its executable face.
• Multi-LoRA serving — the consumer internals we only switch on here: SGMV, unified paging, heterogeneous batching of thousands of adapters.
• Optimising decode in vLLM — how to squeeze the decode throughput of the base you serve the adapters on top of, on a 4090.
• Retrain: closing the feedback → dataset → adapter loop — where the Phase 1 dataset comes from.
• LLM evals: the layer after tracing — how to validate the Phase 4 adapter with judgement, not with the training loss.
• Inverted roofline on small models — the performance regime that explains why the serving bottleneck is the KV cache, not the adapters.
• Aggressive quantisation: from 4-bit to ternary — what happens to the quantised base below NF4 underneath the adapter.

References

• Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
• Hugging Face TRL — PEFT integration (SFTTrainer + QLoRA): https://huggingface.co/docs/trl/peft_integration
• Hugging Face PEFT: https://github.com/huggingface/peft
• bitsandbytes: https://github.com/bitsandbytes-foundation/bitsandbytes
• vLLM — LoRA Adapters (serving, dynamic loading, LoRAResolver): https://docs.vllm.ai/en/stable/features/lora/
• Axolotl: https://github.com/axolotl-ai-cloud/axolotl
• Xu, Y. et al. QA-LoRA: Quantization-Aware Low-Rank Adaptation. ICLR 2024. https://arxiv.org/abs/2309.14717