QLoRA and multi-LoRA at the limit on small models

Contents

This post is the training-side companion to Multi-LoRA serving. That one takes apart the consumer, how hundreds of concurrent adapters are served with SGMV kernels and unified paging; this one takes apart the producer, how an adapter is trained on top of a quantised base on a single GPU, and why the pattern “one frozen base SLM + N low-rank adapters” is the natural fit for small models. We do not repeat the serving internals here; we assume you have read them.

TL;DR

QLoRA (Dettmers et al., NeurIPS 2023) solves a specific problem: fine-tuning a model without having the VRAM to hold its weights in BF16, its gradients and the optimiser states. The idea has three pieces. One: freeze the base and quantise it to 4-bit with a new format, NF4 (NormalFloat 4-bit), quantile-optimal for weights distributed almost like a Gaussian. Two: do not train the base, not a single one of its weights moves, but rather a pair of small LoRA matrices in BF16 plugged in parallel; the gradient flows only through that adapter. Three: two memory tricks, double quantisation (quantising the quantisation constants themselves) and paged optimizers (optimiser states that page out to RAM when VRAM gets tight). The measurable operational result: a 3-8B SLM is fine-tuned on an RTX 4090 (24 GB, Ada Lovelace), not on a cluster. And because the output of training is an adapter measured in megabytes, not gigabytes, the pattern that emerges is a single frozen base SLM in 4-bit plus N adapters, one per customer, domain or task, served on top of the shared base with the stack we already covered in multi-LoRA serving. Per-customer isolation, minimum footprint, sovereign deployment.

The analogy: the frozen guitar and the swappable pedalboard

Think of a session guitarist recording for very different clients: a jazz record, a corporate jingle, a metal track. He has one single guitar, his trusted instrument, set up, with a base tone he knows by heart. What he does not do is buy a new guitar for every song. What he does is keep a pedalboard: a distortion pedal, a chorus, a delay. For each track he plugs in the pedal that fits, and the same guitar sounds completely different.

The mapping is exact:

The guitar = the base SLM. A single copy, set up at the factory, frozen. In QLoRA it is also kept in a compressed case: quantised to 4-bit. You do not touch it: you do not change its pickups or adjust the neck. It weighs what it weighs and there it stays.
Each pedal = a LoRA adapter. Small, cheap, specific to one sound. You train it for a task and put it in a drawer.
Training QLoRA = designing a new pedal by listening to the (frozen) guitar through it, adjusting only the pedal’s pots until it sounds the way you want. The guitar’s base tone is not modified; you learn the correction the pedal applies on top.
Serving multi-LoRA (Multi-LoRA serving) = having the whole pedalboard set up on stage and choosing the right pedal per note, per request. The guitar is the same; what changes between requests is which pedal is active.

The analogy holds down to the detail that confuses people most: the training gradient only “touches” the pedal. The guitar is frozen in its compressed case; learning does not move it. That is what lets the base live in 4-bit throughout the fine-tuning without the quantisation getting in the way: no gradient is ever computed for it.

The bare mechanism: LoRA, and why you can train on a 4-bit base

A minimal reminder of LoRA (Hu et al., ICLR 2022). An adapter modifies a matrix W of the base by adding a low-rank product to it:

$$W' = W + B A, \qquad A \in \mathbb{R}^{r \times d}, \quad B \in \mathbb{R}^{d \times r}$$

with r the rank, much smaller than d. In the forward pass BA is not materialised; what is computed is:

$$y = W x + B(A x)$$

The base compute (Wx) happens just the same; the adapter adds two cheap matmuls. The key to QLoRA is who receives a gradient. The base W is frozen: ∂L/∂W is neither computed nor stored. Only A and B are trainable. That is why W can live quantised to 4-bit without trouble: in the forward pass the quantisation is undone on the fly to compute Wx (dequant → matmul in BF16), but since W is never updated it does not need the precision of a trainable weight. The adapter A, B is in BF16, and it is the only path the gradient flows through.

This is what breaks the memory wall. In a full fine-tuning you need, per weight: the weight (2 bytes BF16), its gradient (2 bytes), and the two Adam states (momentum and variance, typically 4+4 bytes in FP32), on the order of 12-16 bytes per trainable parameter. With QLoRA, the base weights take 0.5 bytes (4-bit) and have no gradient and no optimiser states. Only the few million adapter parameters pay the 16-byte cost. For an 8B, that is the difference between ~130 GB and fitting in 24 GB.

NF4: why a new format instead of INT4

QLoRA does not use linear INT4 for the base, but NF4 (NormalFloat 4-bit). The intuition: the weights of a trained transformer are distributed, empirically, very close to a zero-centred Gaussian. INT4 spreads its 16 levels uniformly across the range, which wastes levels in the tails (where there are almost no weights) and leaves few in the centre (where they pile up). NF4 spreads the 16 levels according to the quantiles of a normal distribution: more levels where there is more probability mass. It is, by construction, information-theoretically optimal for exactly Gaussian data, each level covers roughly the same number of weights. It is also symmetric about zero and guarantees an exact representation of 0 (important for sparsity and padding). The detail of the quantisation formats is in Quantisation for LLM inference; here the idea that NF4 spends its bits where the weights are is enough.

Double quantisation and paged optimizers

Quantising to 4-bit is not entirely free: for each block of weights (typically 64) you need to store a scale constant in FP32 so the quantisation can be undone. Those constants weigh something. With blocks of 64 and one FP32 scale (32 bits) per block, that is 32/64 = 0.5 bits per parameter in metadata alone, a 12.5 % overhead on top of the 4 useful bits. Double quantisation attacks that: it quantises the scale constants themselves (to 8-bit, in blocks of 256), bringing the overhead down to ~0.127 bits/param. Quantising the quantisation sounds recursive and it is; the saving is small in absolute terms (~0.37 bits/param) but on an 8B that is hundreds of MB, which is exactly the margin separating “it fits” from “it does not fit” on a 4090.

Paged optimizers attack the memory peaks. During training, certain moments, a batch with a very long sequence, a large activation, push VRAM close to the limit and blow up with an OOM. The idea, borrowed from operating system paging, is to allocate the optimiser states in NVIDIA unified memory: when VRAM gets tight, those pages are evicted to host RAM automatically and brought back when needed. It speeds nothing up; it avoids the crash at the peaks. It turns an “intermittent OOM” into “a bit slower at the worst moments”, which for an unattended training run on a single GPU is the difference between finishing and not finishing.

“Aggressive” training: very low rank and QA-LoRA

“Aggressive” in this context means two things, sometimes combined.

Very low rank (r = 4-8). The rank is the bottleneck of the correction: how much “capacity” the adapter has to steer the base away from itself. A high rank (64, 128) brings the adapter closer to a full fine-tuning but weighs more and takes longer to train. For an SLM adapted to a narrow, well-defined task, an output format, a lexical domain, a response style, a rank of 4-8 is usually enough, and the resulting adapter weighs a fraction. The risk of a low rank is underfitting: if the task demands rewriting a lot of the base’s behaviour, r=4 falls short. The honest rule is empirical: raise the rank only if the eval asks for it, not “just in case”. On small SLMs, where the base has less spare capacity, a low rank tends to work proportionally better than on large models, but this depends on the task and has to be measured, not assumed.

QA-LoRA (quantization-aware LoRA, Xu et al., arXiv:2309.14717). There is a subtle friction in standard QLoRA: you train the adapter in BF16 against a 4-bit base, but if you then want to merge the adapter into the base (W' = W + BA) to serve a clean quantised model, the merge reintroduces precision that the 4-bit format cannot represent, and on requantising you lose part of what was learnt. QA-LoRA trains the adapter aware of the target’s quantisation: it balances the degrees of freedom of the quantisation and of the adaptation (with group-wise quantisation) so that, when it finishes, the adapter merges cleanly into a quantised base without a requantisation step that degrades it. The result is a final model that is quantised-plus-adapted, with no separate adapter at runtime, useful when you want a single deployable artefact per task instead of the shared-base + adapters pattern. The choice between “QLoRA + serve multi-adapter” and “QA-LoRA + merge per task” is a deployment architecture decision, not one of pure quality.

The maths that matters

Three calculations drive any QLoRA decision on SLMs.

Adapter parameters. For each target matrix of dimension d with rank r, the adapter contributes A (r×d) plus B (d×r), that is 2·r·d parameters. Summing over the target matrices and multiplying by the number of layers:

$$\text{params}_{\text{adapter}} = L \cdot \sum_{\text{matrices}} 2 \cdot r \cdot d$$

Worked example — Llama-3-8B, attention (q, k, v, o), d = 4096, L = 32 layers, r = 8. Taking the four attention projections with the same d = 4096 (a simplification; in Llama-3 K and V are narrower because of GQA, which gives even fewer params):

$$\text{params} \approx 32 \cdot 4 \cdot (2 \cdot 8 \cdot 4096) = 32 \cdot 4 \cdot 65\,536 \approx 8.4\text{M params}$$

In BF16 (2 bytes/param): 8.4M · 2 ≈ 16.8 MB ≈ ~17 MB. Seventeen megabytes. Compare that with the base: an 8B in NF4 takes 8\text{G} · 0.5\,\text{bytes} ≈ 4\text{ GB} (plus the small constants overhead after double quantisation). The adapter is 0.4 % of the size of the quantised base. This is what makes having hundreds operationally trivial: an adapter is not a model, it is almost a heavy configuration file.

How many adapters fit on a 4090 after the base + KV? Budget for an RTX 4090 (24 GB): 8B NF4 base ~4 GB, leave ~5 GB for KV cache and inference activations at moderate concurrency → that leaves ~15 GB free (being conservative, call it ~12-15 GB). With ~17 MB adapters (r=8, attention-only):

$$\frac{15\,000\ \text{MB}}{17\ \text{MB/adapter}} \approx 880 \text{ adapters}$$

On the order of thousands if you cut the reserved KV cache or use rank 4 (~8.5 MB/adapter → ~1750 in 15 GB). The bottleneck is never the adapters’ space; it is the KV cache and concurrency. For the details of how those thousands are served concurrently, the heterogeneous batching, the unified paging, the SGMV kernels, see Multi-LoRA serving. The relevant summary here: the adapter compute is nearly free (low rank, two thin matmuls); the serving performance challenge is not that compute but the gather/scatter of the right adapters per batch row when a single batch mixes requests from different adapters. That is the consumer’s problem, not the producer’s.

QLoRA training VRAM in 24 GB. The rough budget to fine-tune the 8B on a 4090:

Component	Approx. VRAM
8B base in NF4 (frozen weights)	~4.0 GB
Adapter (BF16 params + gradient + Adam states, ~16 B/param over ~8-40M params)	~0.3-0.7 GB
Activations (depends on batch and sequence length; the variable bulk)	~6-14 GB
Dequant buffers, scales, workspace	~1-2 GB
Total	fits in 24 GB with margin

The large, variable piece is the activations, which scale with batch × sequence length. That is why real QLoRA on a 4090 is done with a small batch + gradient accumulation (simulating a large batch by accumulating gradients over microbatches) + gradient checkpointing (recomputing activations in the backward pass instead of storing them, trading compute for memory) + bounded sequences. The paged optimizers are the airbag for the activation peaks that would otherwise blow up. The claim “QLoRA fine-tunes an 8B on a 4090” is true with that configuration; with long sequences, a large batch or a high rank, it does not fit. As with any number, the methodology matters more than the headline.

The fit with small models and sovereignty

This is where QLoRA + SLM stops being a VRAM trick and becomes an architectural pattern.

An SLM (3-8B) already fits comfortably on a single GPU for inference. If on top of that the base lives in 4-bit (~4 GB for an 8B), you have memory to spare. What QLoRA enables is for that same machine, the 4090, to be both the producer and the consumer: you train a new customer’s adapter in hours, on the same class of hardware where you then serve it. The artefact that travels between “train” and “deploy” is an adapter of MB, not GB: it is versioned, signed, moved across the network, stored in MinIO/S3 without thinking about the cost.

The sovereign pattern falls out on its own:

Per-customer isolation. Each customer has their adapter, trained only on their data. The base is generic and shared; what is specific to the customer lives isolated in their (A, B) pair. Deleting a customer is deleting a file of a few MB, not retraining anything.
Minimum footprint. One base + N adapters fits where N bases would not come close. The economics of “one model per customer” (tens of GB each) are prohibitive; those of “one base + adapters” (MB each) are trivial. It is exactly the difference between the pedalboard and buying a guitar per song.
Sovereign deployment. Everything fits on-premise, on your hardware, without a single data item leaving the perimeter. Training (QLoRA on the 4090) and serving (multi-LoRA on the same base) live inside. There is no dependency on an external API for fine-tuning or for serving.

The choice between adapting by domain (one adapter per knowledge area) and retrieving by context (RAG that injects the knowledge into the prompt) is real and not mutually exclusive: the adapter changes the model’s behaviour and style, RAG changes the facts it has access to. The sibling post on aggressive RAG on small models in this series works through it; the short rule is: adapt what is stable and behavioural, retrieve what is volatile and factual.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

This is QLoRA’s natural workbench. The canonical case: a 3-8B base SLM in NF4, fine-tuning an r=8 attention-only adapter, with gradient checkpointing + gradient accumulation + paged optimizer. It trains in hours for narrow-task datasets (thousands to tens of thousands of examples), and the same machine then serves the base + tens or hundreds of adapters for multi-tenant demos and platform prototypes. The 4090 is where QLoRA went from “a paper technique” to “anyone with a consumer GPU can do it”, and that is exactly its value. The honest rule: it fits with the memory configuration described; with long sequences, a large batch or a high rank, move up the hardware.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here QLoRA stops being strictly necessary in order to fit, an 8B in BF16 has room to spare, but it remains useful for another reason: parallelising adapter production. With 320 GB and native FP8 you can train several adapters at once (one job per customer, several in parallel), or fine-tune somewhat larger models with QLoRA without TP. The consumer on this cluster is the serious setup from Multi-LoRA serving: an FP8 base + hundreds of concurrent adapters. The rule of thumb: on the 4090, QLoRA is the tool to be able to fine-tune; on the H100 cluster, it is the tool to fine-tune many at once, cheaply, keeping the quantised format consistent between training and serving.

What we have not covered

The internals of heterogeneous serving (SGMV kernels, MBGMM/MBGMV, unified paging, cold start, eviction): they are covered in full in Multi-LoRA serving. This post is deliberately the producer’s side.
DoRA and variants (magnitude-direction decomposition): they close part of the gap with full fine-tuning; different training pattern, identical serving pattern.
Sub-4-bit and ternary quantisation of the base: what happens when the base drops from NF4 to 2-bit or ternary underneath the adapter; the sibling post in the series works through it.
Collecting the fine-tuning dataset: how each adapter’s corpus is built from production feedback is in Retrain: closing the loop.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. ICLR 2024. https://arxiv.org/abs/2309.14717
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Official QLoRA / bitsandbytes repo: https://github.com/artidoro/qlora
Hugging Face PEFT (LoRA, QLoRA): https://github.com/huggingface/peft