Punica on lo0 — Tech Blog

QLoRA and multi-LoRA at the limit on small models

Tue, 09 Jun 2026 02:30:00 +0000

This post is the training-side companion to Multi-LoRA serving. That one takes apart the consumer, how hundreds of concurrent adapters are served with SGMV kernels and unified paging; this one takes apart the producer, how an adapter is trained on top of a quantised base on a single GPU, and why the pattern “one frozen base SLM + N low-rank adapters” is the natural fit for small models. We do not repeat the serving internals here; we assume you have read them.

TL;DR

QLoRA (Dettmers et al., NeurIPS 2023) solves a specific problem: fine-tuning a model without having the VRAM to hold its weights in BF16, its gradients and the optimiser states. The idea has three pieces. One: freeze the base and quantise it to 4-bit with a new format, NF4 (NormalFloat 4-bit), quantile-optimal for weights distributed almost like a Gaussian. Two: do not train the base, not a single one of its weights moves, but rather a pair of small LoRA matrices in BF16 plugged in parallel; the gradient flows only through that adapter. Three: two memory tricks, double quantisation (quantising the quantisation constants themselves) and paged optimizers (optimiser states that page out to RAM when VRAM gets tight). The measurable operational result: a 3-8B SLM is fine-tuned on an RTX 4090 (24 GB, Ada Lovelace), not on a cluster. And because the output of training is an adapter measured in megabytes, not gigabytes, the pattern that emerges is a single frozen base SLM in 4-bit plus N adapters, one per customer, domain or task, served on top of the shared base with the stack we already covered in multi-LoRA serving. Per-customer isolation, minimum footprint, sovereign deployment.

The analogy: the frozen guitar and the swappable pedalboard

Think of a session guitarist recording for very different clients: a jazz record, a corporate jingle, a metal track. He has one single guitar, his trusted instrument, set up, with a base tone he knows by heart. What he does not do is buy a new guitar for every song. What he does is keep a pedalboard: a distortion pedal, a chorus, a delay. For each track he plugs in the pedal that fits, and the same guitar sounds completely different.

The mapping is exact:

The guitar = the base SLM. A single copy, set up at the factory, frozen. In QLoRA it is also kept in a compressed case: quantised to 4-bit. You do not touch it: you do not change its pickups or adjust the neck. It weighs what it weighs and there it stays.
Each pedal = a LoRA adapter. Small, cheap, specific to one sound. You train it for a task and put it in a drawer.
Training QLoRA = designing a new pedal by listening to the (frozen) guitar through it, adjusting only the pedal’s pots until it sounds the way you want. The guitar’s base tone is not modified; you learn the correction the pedal applies on top.
Serving multi-LoRA (Multi-LoRA serving) = having the whole pedalboard set up on stage and choosing the right pedal per note, per request. The guitar is the same; what changes between requests is which pedal is active.

The analogy holds down to the detail that confuses people most: the training gradient only “touches” the pedal. The guitar is frozen in its compressed case; learning does not move it. That is what lets the base live in 4-bit throughout the fine-tuning without the quantisation getting in the way: no gradient is ever computed for it.

The bare mechanism: LoRA, and why you can train on a 4-bit base

A minimal reminder of LoRA (Hu et al., ICLR 2022). An adapter modifies a matrix W of the base by adding a low-rank product to it:

$$W' = W + B A, \qquad A \in \mathbb{R}^{r \times d}, \quad B \in \mathbb{R}^{d \times r}$$

with r the rank, much smaller than d. In the forward pass BA is not materialised; what is computed is:

$$y = W x + B(A x)$$

The base compute (Wx) happens just the same; the adapter adds two cheap matmuls. The key to QLoRA is who receives a gradient. The base W is frozen: ∂L/∂W is neither computed nor stored. Only A and B are trainable. That is why W can live quantised to 4-bit without trouble: in the forward pass the quantisation is undone on the fly to compute Wx (dequant → matmul in BF16), but since W is never updated it does not need the precision of a trainable weight. The adapter A, B is in BF16, and it is the only path the gradient flows through.

This is what breaks the memory wall. In a full fine-tuning you need, per weight: the weight (2 bytes BF16), its gradient (2 bytes), and the two Adam states (momentum and variance, typically 4+4 bytes in FP32), on the order of 12-16 bytes per trainable parameter. With QLoRA, the base weights take 0.5 bytes (4-bit) and have no gradient and no optimiser states. Only the few million adapter parameters pay the 16-byte cost. For an 8B, that is the difference between ~130 GB and fitting in 24 GB.

NF4: why a new format instead of INT4

QLoRA does not use linear INT4 for the base, but NF4 (NormalFloat 4-bit). The intuition: the weights of a trained transformer are distributed, empirically, very close to a zero-centred Gaussian. INT4 spreads its 16 levels uniformly across the range, which wastes levels in the tails (where there are almost no weights) and leaves few in the centre (where they pile up). NF4 spreads the 16 levels according to the quantiles of a normal distribution: more levels where there is more probability mass. It is, by construction, information-theoretically optimal for exactly Gaussian data, each level covers roughly the same number of weights. It is also symmetric about zero and guarantees an exact representation of 0 (important for sparsity and padding). The detail of the quantisation formats is in Quantisation for LLM inference; here the idea that NF4 spends its bits where the weights are is enough.

Double quantisation and paged optimizers

Quantising to 4-bit is not entirely free: for each block of weights (typically 64) you need to store a scale constant in FP32 so the quantisation can be undone. Those constants weigh something. With blocks of 64 and one FP32 scale (32 bits) per block, that is 32/64 = 0.5 bits per parameter in metadata alone, a 12.5 % overhead on top of the 4 useful bits. Double quantisation attacks that: it quantises the scale constants themselves (to 8-bit, in blocks of 256), bringing the overhead down to ~0.127 bits/param. Quantising the quantisation sounds recursive and it is; the saving is small in absolute terms (~0.37 bits/param) but on an 8B that is hundreds of MB, which is exactly the margin separating “it fits” from “it does not fit” on a 4090.

Paged optimizers attack the memory peaks. During training, certain moments, a batch with a very long sequence, a large activation, push VRAM close to the limit and blow up with an OOM. The idea, borrowed from operating system paging, is to allocate the optimiser states in NVIDIA unified memory: when VRAM gets tight, those pages are evicted to host RAM automatically and brought back when needed. It speeds nothing up; it avoids the crash at the peaks. It turns an “intermittent OOM” into “a bit slower at the worst moments”, which for an unattended training run on a single GPU is the difference between finishing and not finishing.

“Aggressive” training: very low rank and QA-LoRA

“Aggressive” in this context means two things, sometimes combined.

Very low rank (r = 4-8). The rank is the bottleneck of the correction: how much “capacity” the adapter has to steer the base away from itself. A high rank (64, 128) brings the adapter closer to a full fine-tuning but weighs more and takes longer to train. For an SLM adapted to a narrow, well-defined task, an output format, a lexical domain, a response style, a rank of 4-8 is usually enough, and the resulting adapter weighs a fraction. The risk of a low rank is underfitting: if the task demands rewriting a lot of the base’s behaviour, r=4 falls short. The honest rule is empirical: raise the rank only if the eval asks for it, not “just in case”. On small SLMs, where the base has less spare capacity, a low rank tends to work proportionally better than on large models, but this depends on the task and has to be measured, not assumed.

QA-LoRA (quantization-aware LoRA, Xu et al., arXiv:2309.14717). There is a subtle friction in standard QLoRA: you train the adapter in BF16 against a 4-bit base, but if you then want to merge the adapter into the base (W' = W + BA) to serve a clean quantised model, the merge reintroduces precision that the 4-bit format cannot represent, and on requantising you lose part of what was learnt. QA-LoRA trains the adapter aware of the target’s quantisation: it balances the degrees of freedom of the quantisation and of the adaptation (with group-wise quantisation) so that, when it finishes, the adapter merges cleanly into a quantised base without a requantisation step that degrades it. The result is a final model that is quantised-plus-adapted, with no separate adapter at runtime, useful when you want a single deployable artefact per task instead of the shared-base + adapters pattern. The choice between “QLoRA + serve multi-adapter” and “QA-LoRA + merge per task” is a deployment architecture decision, not one of pure quality.

The maths that matters

Three calculations drive any QLoRA decision on SLMs.

Adapter parameters. For each target matrix of dimension d with rank r, the adapter contributes A (r×d) plus B (d×r), that is 2·r·d parameters. Summing over the target matrices and multiplying by the number of layers:

$$\text{params}_{\text{adapter}} = L \cdot \sum_{\text{matrices}} 2 \cdot r \cdot d$$

Worked example — Llama-3-8B, attention (q, k, v, o), d = 4096, L = 32 layers, r = 8. Taking the four attention projections with the same d = 4096 (a simplification; in Llama-3 K and V are narrower because of GQA, which gives even fewer params):

$$\text{params} \approx 32 \cdot 4 \cdot (2 \cdot 8 \cdot 4096) = 32 \cdot 4 \cdot 65\,536 \approx 8.4\text{M params}$$

In BF16 (2 bytes/param): 8.4M · 2 ≈ 16.8 MB ≈ ~17 MB. Seventeen megabytes. Compare that with the base: an 8B in NF4 takes 8\text{G} · 0.5\,\text{bytes} ≈ 4\text{ GB} (plus the small constants overhead after double quantisation). The adapter is 0.4 % of the size of the quantised base. This is what makes having hundreds operationally trivial: an adapter is not a model, it is almost a heavy configuration file.

How many adapters fit on a 4090 after the base + KV? Budget for an RTX 4090 (24 GB): 8B NF4 base ~4 GB, leave ~5 GB for KV cache and inference activations at moderate concurrency → that leaves ~15 GB free (being conservative, call it ~12-15 GB). With ~17 MB adapters (r=8, attention-only):

$$\frac{15\,000\ \text{MB}}{17\ \text{MB/adapter}} \approx 880 \text{ adapters}$$

On the order of thousands if you cut the reserved KV cache or use rank 4 (~8.5 MB/adapter → ~1750 in 15 GB). The bottleneck is never the adapters’ space; it is the KV cache and concurrency. For the details of how those thousands are served concurrently, the heterogeneous batching, the unified paging, the SGMV kernels, see Multi-LoRA serving. The relevant summary here: the adapter compute is nearly free (low rank, two thin matmuls); the serving performance challenge is not that compute but the gather/scatter of the right adapters per batch row when a single batch mixes requests from different adapters. That is the consumer’s problem, not the producer’s.

QLoRA training VRAM in 24 GB. The rough budget to fine-tune the 8B on a 4090:

Component	Approx. VRAM
8B base in NF4 (frozen weights)	~4.0 GB
Adapter (BF16 params + gradient + Adam states, ~16 B/param over ~8-40M params)	~0.3-0.7 GB
Activations (depends on batch and sequence length; the variable bulk)	~6-14 GB
Dequant buffers, scales, workspace	~1-2 GB
Total	fits in 24 GB with margin

The large, variable piece is the activations, which scale with batch × sequence length. That is why real QLoRA on a 4090 is done with a small batch + gradient accumulation (simulating a large batch by accumulating gradients over microbatches) + gradient checkpointing (recomputing activations in the backward pass instead of storing them, trading compute for memory) + bounded sequences. The paged optimizers are the airbag for the activation peaks that would otherwise blow up. The claim “QLoRA fine-tunes an 8B on a 4090” is true with that configuration; with long sequences, a large batch or a high rank, it does not fit. As with any number, the methodology matters more than the headline.

The fit with small models and sovereignty

This is where QLoRA + SLM stops being a VRAM trick and becomes an architectural pattern.

An SLM (3-8B) already fits comfortably on a single GPU for inference. If on top of that the base lives in 4-bit (~4 GB for an 8B), you have memory to spare. What QLoRA enables is for that same machine, the 4090, to be both the producer and the consumer: you train a new customer’s adapter in hours, on the same class of hardware where you then serve it. The artefact that travels between “train” and “deploy” is an adapter of MB, not GB: it is versioned, signed, moved across the network, stored in MinIO/S3 without thinking about the cost.

The sovereign pattern falls out on its own:

Per-customer isolation. Each customer has their adapter, trained only on their data. The base is generic and shared; what is specific to the customer lives isolated in their (A, B) pair. Deleting a customer is deleting a file of a few MB, not retraining anything.
Minimum footprint. One base + N adapters fits where N bases would not come close. The economics of “one model per customer” (tens of GB each) are prohibitive; those of “one base + adapters” (MB each) are trivial. It is exactly the difference between the pedalboard and buying a guitar per song.
Sovereign deployment. Everything fits on-premise, on your hardware, without a single data item leaving the perimeter. Training (QLoRA on the 4090) and serving (multi-LoRA on the same base) live inside. There is no dependency on an external API for fine-tuning or for serving.

The choice between adapting by domain (one adapter per knowledge area) and retrieving by context (RAG that injects the knowledge into the prompt) is real and not mutually exclusive: the adapter changes the model’s behaviour and style, RAG changes the facts it has access to. The sibling post on aggressive RAG on small models in this series works through it; the short rule is: adapt what is stable and behavioural, retrieve what is volatile and factual.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

This is QLoRA’s natural workbench. The canonical case: a 3-8B base SLM in NF4, fine-tuning an r=8 attention-only adapter, with gradient checkpointing + gradient accumulation + paged optimizer. It trains in hours for narrow-task datasets (thousands to tens of thousands of examples), and the same machine then serves the base + tens or hundreds of adapters for multi-tenant demos and platform prototypes. The 4090 is where QLoRA went from “a paper technique” to “anyone with a consumer GPU can do it”, and that is exactly its value. The honest rule: it fits with the memory configuration described; with long sequences, a large batch or a high rank, move up the hardware.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here QLoRA stops being strictly necessary in order to fit, an 8B in BF16 has room to spare, but it remains useful for another reason: parallelising adapter production. With 320 GB and native FP8 you can train several adapters at once (one job per customer, several in parallel), or fine-tune somewhat larger models with QLoRA without TP. The consumer on this cluster is the serious setup from Multi-LoRA serving: an FP8 base + hundreds of concurrent adapters. The rule of thumb: on the 4090, QLoRA is the tool to be able to fine-tune; on the H100 cluster, it is the tool to fine-tune many at once, cheaply, keeping the quantised format consistent between training and serving.

What we have not covered

The internals of heterogeneous serving (SGMV kernels, MBGMM/MBGMV, unified paging, cold start, eviction): they are covered in full in Multi-LoRA serving. This post is deliberately the producer’s side.
DoRA and variants (magnitude-direction decomposition): they close part of the gap with full fine-tuning; different training pattern, identical serving pattern.
Sub-4-bit and ternary quantisation of the base: what happens when the base drops from NF4 to 2-bit or ternary underneath the adapter; the sibling post in the series works through it.
Collecting the fine-tuning dataset: how each adapter’s corpus is built from production feedback is in Retrain: closing the loop.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. ICLR 2024. https://arxiv.org/abs/2309.14717
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Official QLoRA / bitsandbytes repo: https://github.com/artidoro/qlora
Hugging Face PEFT (LoRA, QLoRA): https://github.com/huggingface/peft

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

Sat, 30 May 2026 14:00:00 +0200

This post complements Continuous fine-tuning in production. Continuous fine-tuning is the producer of the adapters; multi-LoRA serving is the consumer that puts them to work. Without this layer, the whole feedback cycle breaks in the last mile. It also crosses over with Modern alignment (DPO/KTO/ORPO/SimPO) (each alignment policy can live as a separate adapter) and Quantisation (a quantised base frees memory for many more adapters).

You are here: DEPLOY

TL;DR

The dominant pattern in 2026 is not one model per customer but a single general-purpose base model plus N fine LoRA adapters per task, customer, language or domain. The reason is obvious: a rank-16 LoRA over Llama-3-70B takes about 400 MB; a full fine-tune takes about 140 GB. Dozens or hundreds of adapters per base is manageable; dozens or hundreds of bases is prohibitive. What is not obvious is how to serve them concurrently without reloading weights every time the adapter changes (killing batching) or replicating the base (killing memory). The answer crystallised in 2024 with two complementary papers: S-LoRA (Sheng et al., Stanford + UC Berkeley, MLSys 2024) introduced unified paging, where adapter weights live in the same memory pool as the KV cache, both pageable, and heterogeneous batching, where a batch can hold requests with different adapters and different ranks without padding; Punica (Chen et al., UW + Duke, MLSys 2024) introduced the CUDA kernel that has become the de facto standard: SGMV (Segmented Gather Matrix-Vector multiplication), which computes Y += Σ_i X_i · A_i · B_i in a single pass, grouping requests by adapter. SGMV sits today underneath vLLM, LoRAX (Predibase), SGLang and TGI. The measurable operational result: up to 2,000 concurrent adapters on a single GPU (S-LoRA paper), up to 4× throughput vs naive vLLM and up to 30× vs HuggingFace PEFT. The price: typical overhead of 10-30 % latency per layer with an active adapter in a heterogeneous batch, practically zero when every request in the batch uses the same adapter, and 20-40 % in the worst case. This post takes apart the mechanism, the maths (memory per adapter, overhead per rank), the comparison table of implementations, the pitfalls (cold start, mismatched rank, fragmentation) and the real economics on an H100 with a Llama-3-70B FP8 base + 200 adapters.

The analogy: the single translator with a thousand glossaries

Imagine a specialised translation agency with a single senior translator, brilliant, fluent in fifteen languages and across all the general technical domains. That translator is expensive to hire and expensive to train: it took years of formation and experience that is not easily replicated. But the agency receives texts from very different clients: a law firm using legal terminology specific to its jurisdiction, a manufacturer with an internal parts nomenclature, a hospital with its own clinical abbreviations. Every client has its own jargon.

The agency does not hire one translator per client; that would be ridiculous, 90 % of the work is shared. What it does is keep one glossary per client: a small notebook, easy to update, containing the specific terms and how they are translated for that client. When the translator receives a text, they open the glossary for the client in question and work with it alongside. As they translate each word, they first check whether it is in the glossary; if it is, they use the specific version; if not, they use their general knowledge.

The glossaries live on a shared shelf, ordered by recent use: the most consulted ones within reach, the old ones in the archive. When a new client arrives, their glossary is brought from the archive to the shelf. When the desk fills up, the least used glossary goes back to the archive.

And the most important part: the translator can have several glossaries open at once because they are working in parallel on five texts from five clients. It is not one glossary per document; it is one glossary per client, and client A’s documents use A’s glossary, client B’s use B’s, all on the same desk.

The analogy holds on five mappings:

The single translator = the base model (Llama-3-70B, Qwen2.5-72B). Expensive to train, a single copy in VRAM.
Each glossary = a LoRA adapter. Small (~150-400 MB), specific, easy to update.
The shelf with the glossaries within reach = the pool of adapters cached in VRAM (typically 50-200 at a time with an FP8 base on an H100 80 GB).
The archive = adapter storage on MinIO/S3/HF Hub. Hundreds or thousands, fetched on demand.
Working in parallel with several glossaries open = a heterogeneous batch with SGMV. The kernel does the grouped lookup into the right glossary for each word in the batch.

The bare mechanism: what a LoRA does and why it can be served multi-tenant

A LoRA adapter modifies a matrix W of the base model by adding a low-rank product to it:

$$W' = W + B A$$

where W ∈ R^{d_out × d_in} is the original matrix (the base weights), A ∈ R^{r × d_in} and B ∈ R^{d_out × r} are the adapter’s trainable matrices, and r is the rank (typically 8, 16, 32 or 64, always much smaller than d_in and d_out).

In a forward pass, instead of computing y = W' x, it computes:

$$y = W x + B(Ax)$$

That is: the base computation (Wx) happens exactly as before; the adapter adds two cheap matmuls (Ax and then B(·)) that supply the correction. The matrix BA is never explicitly materialised.

What this enables in serving: if you have the base loaded and N different adapters, Wx is computed only once for all the tokens in the batch (the base is the same). What changes between tokens is only the delta B_i(A_i x). If the tokens in the batch use different adapters, different deltas have to be applied per token, and that is what the SGMV kernel does in a single pass.

Without a specialised kernel this degenerates: you need to launch N separate matmuls (one per adapter), pay kernel launch overhead N times and lose the batching. With a specialised kernel (SGMV), every delta is computed in a single pass grouped by adapter.

SGMV: the kernel that holds it all up

SGMV (Segmented Gather Matrix-Vector multiplication) is the CUDA kernel Punica introduced and that vLLM, LoRAX, SGLang and TGI have adopted as their multi-LoRA engine.

Its job is to compute, given a batch of tokens with mixed adapters:

$$y_t = W x_t + B_{a(t)} A_{a(t)} x_t \quad \forall t \in \text{batch}$$

where a(t) is the adapter assigned to token t. SGMV operates in two phases:

SGMV-shrink: projection d_in → r with the corresponding A_{a(t)} matrix.
SGMV-expand: projection r → d_out with the corresponding B_{a(t)} matrix.

Internally, SGMV groups the batch’s tokens by adapter (it segments them), and for each segment it uses the optimal kernel for the size: for large segments (several requests with the same adapter) it goes through tensor cores; for small segments (one request per adapter) it uses the batched gather path that minimises launch overhead.

The result, in a single kernel pass, is the correct delta for every token in the batch, whatever its rank or its adapter. Punica reported up to 12× throughput vs vLLM/FasterTransformer/HF Transformers/DeepSpeed in heterogeneous multi-tenant scenarios; when every request uses the same adapter, SGMV is practically equivalent to the base without LoRA because it reduces to the “single large segment” case that is optimal for tensor cores.

S-LoRA refined SGMV with two kernels specific to different serving phases: MBGMM (Multi-size Batched Gather Matrix-Matrix) for prefill, MBGMV for decode. Both support different ranks among requests in the same batch, which was a limitation in the original SGMV.

The maths that matters

Three numbers drive every operational decision with multi-LoRA.

Memory per adapter. For a d_in × d_out matrix with rank r and b bytes per parameter (BF16/FP16 = 2):

$$\text{bytes per matrix} = (d_{\text{in}} \cdot r + r \cdot d_{\text{out}}) \cdot b$$

Summing over all the target matrices in each layer and multiplying by the number of layers gives the adapter size. A concrete calculation for Llama-3-70B, rank 16, BF16, all the matrices (Q, K, V, O, gate, up, down):

Matrix	Dimension	Bytes
Q (8192→8192)	8192·16 + 16·8192	524,288
O (8192→8192)	8192·16 + 16·8192	524,288
K (8192→1024)	8192·16 + 16·1024	294,912
V (8192→1024)	8192·16 + 16·1024	294,912
gate (8192→28,672)	8192·16 + 16·28,672	1,179,648
up (8192→28,672)	8192·16 + 16·28,672	1,179,648
down (28,672→8192)	28,672·16 + 16·8192	1,179,648
Sum per layer		5,177,344 ≈ 4.94 MB
Adapter total (80 layers)		~395 MB

If it is limited to attention-only (Q, O, V, K): about 125 MB per adapter. The choice of which matrices get LoRA belongs to whoever trains it; in serving it is inherited and it determines the cost.

Memory per rank. Linear: rank 8 → ~200 MB; rank 16 → ~400 MB; rank 32 → ~800 MB; rank 64 → ~1.6 GB. The simple sizing rule is: max_lora_rank should be the maximum rank you are going to serve, no more. Setting it higher wastes reserved memory in every slot.

How many adapters fit. For an H100 SXM 80 GB with a Llama-3-70B FP8 base (~70 GB), about 10 GB are left after a minimal KV cache → around 25 fully targeted r=16 adapters or around 80 attention-only. With an AWQ INT4 base (~35 GB), about 45 GB are left → hundreds of adapters. The rule: quantising the base does not just free memory, it multiplies the economics of the platform.

Latency overhead per adapter. Under real reported conditions:

Case	Typical overhead
All requests in the batch on the same adapter	~0 % (equivalent to a static merge)
Heterogeneous batch, similar ranks (e.g. all r=16)	10-30 % per layer
Heterogeneous batch, mismatched ranks (r=8 with r=128)	up to +84 % P95 TTFT for the lower rank
Naive PEFT LoRA (no SGMV)	250-950 % extra

It scales linearly with rank: rank 8 ≈ baseline; rank 64 ≈ 3-4 × overhead.

The real implementations in May 2026

Implementation	Base kernel	Hot-swap	Quantised base + LoRA	Operational notes
vLLM	SGMV + ext	Yes (LoRAResolver, S3/HF/FS plugins)	AWQ/GPTQ yes, bnb 4-bit offline only	The de facto default. `--enable-lora --max-loras N --max-lora-rank R --max-cpu-loras M`
LoRAX (Predibase)	Optimised SGMV	Yes (dynamic loading)	Yes	Designed specifically for multi-LoRA. Supports Medusa adapters per adapter (spec-dec per adapter).
SGLang	SGMV / csgmv	Yes	Yes	`--enable-lora-overlap-loading` cuts TTFT by up to 78 % on LoRA-heavy workloads
TensorRT-LLM	LoRA Executor (C++)	Pre-compile at build time	INT4 + LoRA common	Peak throughput on H100/B200, less flexible than vLLM
HF TGI	Punica fork	Yes (`LORA_ADAPTERS=...`)	Yes	In maintenance mode as of May 2026; HF recommends vLLM or SGLang
NVIDIA NIM	TRT-LLM under the hood	Static or dynamic (`NIM_PEFT_REFRESH_INTERVAL`)	Yes	Adapter store per model; polling for hot-add/remove

Three operational observations:

vLLM dominates open-source serving in 2026 thanks to the combination of a mature SGMV + LoRAResolver plugins + support for a quantised base. The critical parameter is --max-lora-rank: many installations set it to the maximum “just in case” and waste memory silently.
LoRAX wins in production operations with thousands of rarely used adapters thanks to its dynamic loading that does not block concurrent requests. A public case: Convirza with 60+ concurrent adapters and sub-2s P95.
SGLang wins on latency when cold starts are frequent thanks to --enable-lora-overlap-loading (async H2D during the compute of the previous request).

Combined pattern with quantisation and disaggregated serving

With quantisation (Quantisation). The canonical stack in May 2026 is base in FP8 (Hopper/Blackwell) or INT4 AWQ + adapters in BF16/FP16. The adapters are not quantised: they are small, the memory saved is irrelevant, and quantisation noise would accumulate badly with the delta. Quantising the base frees massive amounts of memory for more adapters with no significant loss (<1 % in typical MMLU with AWQ INT4, somewhat more in math/code/reasoning). An adapter trained with a BF16 base works with an FP8/INT4 base at inference with marginal loss, and that is what makes QLoRA operationally trivial: train with a 4-bit base and deploy with a consistent 4-bit base.

With disaggregated serving (Disaggregated serving). Multi-LoRA + prefill/decode disaggregation adds a management layer: each pod needs access to the request’s active adapter. The 2026 strategy: replicate hot adapters across every pod (prefill and decode), evict the cold ones. Cold adapters not present in the target pod are transferred on demand, accepting the extra cost in TTFT. Recent work (InfiniLoRA, FASTLIBRA, LoRAServe) automates this balancing, but the simple rule of thumb works in most deployments.

Operational pitfalls

Cold start. The first request to a sleeping adapter involves a fetch (from S3/MinIO/HF Hub) → CPU load → H2D copy. For an adapter of around 400 MB: 0.5-5 s typically, depending on bandwidth. Under high concurrency it can be up to 35 % of E2E latency (Predictive-LoRA paper, arXiv:2512.20210). Established mitigations: SGLang --enable-lora-overlap-loading (cuts TTFT by 35-78 %); vLLM pre-warming with a dummy request when an adapter is registered; predictive prefetching based on patterns (Predictive-LoRA cuts cold start by 68 %).

Heterogeneous worst case. If every request in the batch has a different adapter and a different rank, SGMV loses its advantage because each segment has size 1. Throughput can fall by up to 50 % vs the base without LoRA. Practical mitigation: group adapters by rank in the upstream routing, trying to get requests of the same rank into the same batch step.

Mismatched rank. Co-batching rank 8 with rank 128 penalises the smaller one: +84 % P95 TTFT for the rank-8 requests (Serving Heterogeneous LoRA Adapters paper, arXiv:2511.22880). Practice: normalise the rank across the fleet whenever possible (train every adapter at the same rank, or at least within a narrow range).

Eviction. LRU is the default. If you have more adapters than fit in CPU RAM, the evicted ones get fetched again from cold storage. Monitoring cold_starts_per_minute and cache_hit_ratio per endpoint is basic hygiene.

Base versioning. Every adapter is tied to an exact base version (Llama-3-70B-Instruct ≠ Llama-3.1-70B-Instruct). The routing must validate the base+adapter pair before serving, or the output will be silent rubbish.

Memory fragmentation. Without paged management (the pre-S-LoRA / pre-vLLM case), evicting and inserting adapters fragments HBM until it becomes unusable. Unified Paging solves it: the LoRA weights and the KV cache live in the same pool of blocks, interchangeable.

Implications on on-premise hardware

On an RTX 4090 (24 GB). The classic case: base Llama-3-8B FP8 (~8 GB) or Llama-3-8B AWQ-INT4 (~5 GB) + dozens of r=16 adapters (~80 MB each, around 3 MB per adapter for Llama-3-8B with r=16 attention-only). 50-100 active adapters fit easily on the 4090. It is the natural setup for multi-tenant demos, fine-tuning over an 8B base and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). This is where the serious setup comes in:

Llama-3-70B FP8 (~70 GB) fits on 2 H100s; the other 2 GPUs are available for batch + adapters.
Llama-3-70B AWQ-INT4 (~35 GB) fits on 1 H100; the rest of the cluster serves more concurrency or more adapters.
~200 fully targeted r=16 adapters fit with a comfortable budget on the cluster, enough for a SaaS platform with dozens of tenants and simultaneous A/B testing.
QLoRA training + consistent serving: the train→deploy adapter cycle takes hours, not days, because the adapter is around 400 MB instead of around 140 GB.

The rule of thumb on an H100 cluster in May 2026: an FP8 or INT4 base + 100-500 adapters per cluster is operationally trivial with vLLM or LoRAX; going past a thousand concurrent adapters starts to require serious tuning of eviction and prefetch.

Typical production stack

[API Gateway]
↓ (JWT with tenant_id / API key)
[Router]
↓ injects adapter_id into the request
[vLLM / LoRAX with --enable-lora]
--max-loras 16
--max-lora-rank 32
--max-cpu-loras 200
+ LoRAResolver → s3://adapters/{tenant_id}/{version}/
+ Base: Llama-3-70B-FP8 loaded once on 4×H100 TP=4
↓
[GPU]
FP8 base (70 GB) + ~150 BF16 adapters hot in HBM + ~1,000 warm in RAM
[MinIO / S3]
Cold storage for thousands of adapters
[CI pipeline]
→ trains new QLoRA adapter → push MinIO → notify server → warm-up
[Observability]
Prom: active_adapters, cache_hit_ratio, cold_starts_per_minute,
per_adapter_throughput, P99_with_lora vs P99_base

Routing pattern: Authorization: Bearer <key> → middleware extracts tenant_id → maps to adapter_id (Postgres or Redis) → POST /v1/completions with model: "<adapter_id>".

What we have not covered

DoRA (Weight-Decomposed LoRA): decomposes the update into magnitude + direction, closing part of the quality gap with full fine-tuning. Supported by TensorRT-LLM and others, but the serving pattern is identical to LoRA.
MoE + LoRA: how adapter fine-tuning is done over an MoE, what happens to the routing. Not trivial, an active research area in 2026.
Activated LoRA (arXiv:2512.17910): a variant that reuses the KV cache between compatible adapters, cutting the cost of cold start with shared prefixes.
LoRA for speculative decoding: each adapter brings its own Medusa head, supported by LoRAX as “Turbo LoRA”.
Compress-then-Serve (arXiv:2407.00066): quantising the adapters themselves to serve even more concurrently. Still a marginal practice in production as of May 2026.

References

Hu, E. et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Dettmers, T. et al. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Brüel-Gabrielsson, R. et al. Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead. 2024. https://arxiv.org/abs/2407.00066
Serving Heterogeneous LoRA Adapters in Distributed LLM Inference. 2025. https://arxiv.org/abs/2511.22880
InfiniLoRA: Disaggregated Multi-LoRA Serving. 2026. https://arxiv.org/abs/2604.07173
Predictive-LoRA: Proactive Fragmentation-Aware Serverless LoRA Serving. https://arxiv.org/abs/2512.20210
S-LoRA repo: https://github.com/S-LoRA/S-LoRA
Punica repo: https://github.com/punica-ai/punica
LoRAX repo (Predibase): https://github.com/predibase/lorax
vLLM LoRA docs: https://docs.vllm.ai/en/stable/features/lora/
SGLang LoRA docs: https://docs.sglang.io/advanced_features/lora.html
TensorRT-LLM LoRA docs: https://nvidia.github.io/TensorRT-LLM/advanced/lora.html
NVIDIA NIM PEFT: https://docs.nvidia.com/nim/large-language-models/latest/peft.html
HF TGI Multi-LoRA blog: https://huggingface.co/blog/multi-lora-serving
LMSYS S-LoRA recipe blog: https://www.lmsys.org/blog/2023-11-15-slora/
Predibase LoRAX blog: https://predibase.com/blog/lorax-the-open-source-framework-for-serving-100s-of-fine-tuned-llms-in

Punica on lo0 — Tech Blog

QLoRA and multi-LoRA at the limit on small models

TL;DR

The analogy: the frozen guitar and the swappable pedalboard

The bare mechanism: LoRA, and why you can train on a 4-bit base

NF4: why a new format instead of INT4

Double quantisation and paged optimizers

“Aggressive” training: very low rank and QA-LoRA

The maths that matters

The fit with small models and sovereignty

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

You are here: DEPLOY

TL;DR

The analogy: the single translator with a thousand glossaries

The bare mechanism: what a LoRA does and why it can be served multi-tenant

SGMV: the kernel that holds it all up

The maths that matters

The real implementations in May 2026

Combined pattern with quantisation and disaggregated serving

Operational pitfalls

Implications on on-premise hardware

Typical production stack

What we have not covered

See also

References