Multi-LoRA serving: the single translator with a thousand glossaries — shared base, thousands of concurrent adapters and the SGMV kernel

· By José Manuel Román
Contents

This post complements Continuous fine-tuning in production. Continuous fine-tuning is the producer of the adapters; multi-LoRA serving is the consumer that puts them to work. Without this layer, the whole feedback cycle breaks in the last mile. It also crosses over with Modern alignment (DPO/KTO/ORPO/SimPO) (each alignment policy can live as a separate adapter) and Quantisation (a quantised base frees memory for many more adapters).

You are here: DEPLOY

You are here: DEPLOY · shared base, N concurrent adapters on a single GPU1 · Data2 · Tune3 · Eval4 · Deploy5 · Observe6 · Retrain

TL;DR

The dominant pattern in 2026 is not one model per customer but a single general-purpose base model plus N fine LoRA adapters per task, customer, language or domain. The reason is obvious: a rank-16 LoRA over Llama-3-70B takes about 400 MB; a full fine-tune takes about 140 GB. Dozens or hundreds of adapters per base is manageable; dozens or hundreds of bases is prohibitive. What is not obvious is how to serve them concurrently without reloading weights every time the adapter changes (killing batching) or replicating the base (killing memory). The answer crystallised in 2024 with two complementary papers: S-LoRA (Sheng et al., Stanford + UC Berkeley, MLSys 2024) introduced unified paging, where adapter weights live in the same memory pool as the KV cache, both pageable, and heterogeneous batching, where a batch can hold requests with different adapters and different ranks without padding; Punica (Chen et al., UW + Duke, MLSys 2024) introduced the CUDA kernel that has become the de facto standard: SGMV (Segmented Gather Matrix-Vector multiplication), which computes Y += Σ_i X_i · A_i · B_i in a single pass, grouping requests by adapter. SGMV sits today underneath vLLM, LoRAX (Predibase), SGLang and TGI. The measurable operational result: up to 2,000 concurrent adapters on a single GPU (S-LoRA paper), up to 4× throughput vs naive vLLM and up to 30× vs HuggingFace PEFT. The price: typical overhead of 10-30 % latency per layer with an active adapter in a heterogeneous batch, practically zero when every request in the batch uses the same adapter, and 20-40 % in the worst case. This post takes apart the mechanism, the maths (memory per adapter, overhead per rank), the comparison table of implementations, the pitfalls (cold start, mismatched rank, fragmentation) and the real economics on an H100 with a Llama-3-70B FP8 base + 200 adapters.

The analogy: the single translator with a thousand glossaries

Imagine a specialised translation agency with a single senior translator, brilliant, fluent in fifteen languages and across all the general technical domains. That translator is expensive to hire and expensive to train: it took years of formation and experience that is not easily replicated. But the agency receives texts from very different clients: a law firm using legal terminology specific to its jurisdiction, a manufacturer with an internal parts nomenclature, a hospital with its own clinical abbreviations. Every client has its own jargon.

The agency does not hire one translator per client; that would be ridiculous, 90 % of the work is shared. What it does is keep one glossary per client: a small notebook, easy to update, containing the specific terms and how they are translated for that client. When the translator receives a text, they open the glossary for the client in question and work with it alongside. As they translate each word, they first check whether it is in the glossary; if it is, they use the specific version; if not, they use their general knowledge.

The glossaries live on a shared shelf, ordered by recent use: the most consulted ones within reach, the old ones in the archive. When a new client arrives, their glossary is brought from the archive to the shelf. When the desk fills up, the least used glossary goes back to the archive.

And the most important part: the translator can have several glossaries open at once because they are working in parallel on five texts from five clients. It is not one glossary per document; it is one glossary per client, and client A’s documents use A’s glossary, client B’s use B’s, all on the same desk.

The analogy holds on five mappings:

  • The single translator = the base model (Llama-3-70B, Qwen2.5-72B). Expensive to train, a single copy in VRAM.
  • Each glossary = a LoRA adapter. Small (~150-400 MB), specific, easy to update.
  • The shelf with the glossaries within reach = the pool of adapters cached in VRAM (typically 50-200 at a time with an FP8 base on an H100 80 GB).
  • The archive = adapter storage on MinIO/S3/HF Hub. Hundreds or thousands, fetched on demand.
  • Working in parallel with several glossaries open = a heterogeneous batch with SGMV. The kernel does the grouped lookup into the right glossary for each word in the batch.

The bare mechanism: what a LoRA does and why it can be served multi-tenant

A LoRA adapter modifies a matrix W of the base model by adding a low-rank product to it:

$$W' = W + B A$$

where W ∈ R^{d_out × d_in} is the original matrix (the base weights), A ∈ R^{r × d_in} and B ∈ R^{d_out × r} are the adapter’s trainable matrices, and r is the rank (typically 8, 16, 32 or 64, always much smaller than d_in and d_out).

In a forward pass, instead of computing y = W' x, it computes:

$$y = W x + B(Ax)$$

That is: the base computation (Wx) happens exactly as before; the adapter adds two cheap matmuls (Ax and then B(·)) that supply the correction. The matrix BA is never explicitly materialised.

What this enables in serving: if you have the base loaded and N different adapters, Wx is computed only once for all the tokens in the batch (the base is the same). What changes between tokens is only the delta B_i(A_i x). If the tokens in the batch use different adapters, different deltas have to be applied per token, and that is what the SGMV kernel does in a single pass.

Without a specialised kernel this degenerates: you need to launch N separate matmuls (one per adapter), pay kernel launch overhead N times and lose the batching. With a specialised kernel (SGMV), every delta is computed in a single pass grouped by adapter.

SGMV: the kernel that holds it all up

SGMV (Segmented Gather Matrix-Vector multiplication) is the CUDA kernel Punica introduced and that vLLM, LoRAX, SGLang and TGI have adopted as their multi-LoRA engine.

Its job is to compute, given a batch of tokens with mixed adapters:

$$y_t = W x_t + B_{a(t)} A_{a(t)} x_t \quad \forall t \in \text{batch}$$

where a(t) is the adapter assigned to token t. SGMV operates in two phases:

  1. SGMV-shrink: projection d_in → r with the corresponding A_{a(t)} matrix.
  2. SGMV-expand: projection r → d_out with the corresponding B_{a(t)} matrix.

Internally, SGMV groups the batch’s tokens by adapter (it segments them), and for each segment it uses the optimal kernel for the size: for large segments (several requests with the same adapter) it goes through tensor cores; for small segments (one request per adapter) it uses the batched gather path that minimises launch overhead.

The result, in a single kernel pass, is the correct delta for every token in the batch, whatever its rank or its adapter. Punica reported up to 12× throughput vs vLLM/FasterTransformer/HF Transformers/DeepSpeed in heterogeneous multi-tenant scenarios; when every request uses the same adapter, SGMV is practically equivalent to the base without LoRA because it reduces to the “single large segment” case that is optimal for tensor cores.

S-LoRA refined SGMV with two kernels specific to different serving phases: MBGMM (Multi-size Batched Gather Matrix-Matrix) for prefill, MBGMV for decode. Both support different ranks among requests in the same batch, which was a limitation in the original SGMV.

1. Heterogeneous batch: four requests with three different adaptersreq_1 → A12tenant_1req_2 → A12tenant_1 (another chat)req_3 → A47tenant_2req_4 → A89tenant_32. SGMV kernel: groups by adapter, computes in a single passSGMV: Y = Wx (base) + Σ_a B_a · A_a · x_{tokens(a)}segment A12 (2 reqs, rank=16) | segment A47 (1 req, rank=8) | segment A89 (1 req, rank=32)tensor cores for the large segment, batched gather for the small ones3. GPU memory: shared base + unified adapter pool + KV cacheBASE: Llama-3-70B FP8 (~70 GB) — loaded once, shared by allcomputes Wx for every token in the batch whatever the adapterHBM POOL: ~10 GB free~25 active r=16 adapters (hot)A12, A47, A89, A03, A18, A23, ...RAM CACHE: ~512 GB~1,300 warm adaptersLRU eviction; async H2D on use4. STORAGEMinIO / S3 / HF Hubcold storage: thousands of adaptersA0001 ... A9999cold start: ~0.5-5s per adapterSolid arrow = on-demand path (cache miss). Dashed = LRU eviction.

The maths that matters

Three numbers drive every operational decision with multi-LoRA.

Memory per adapter. For a d_in × d_out matrix with rank r and b bytes per parameter (BF16/FP16 = 2):

$$\text{bytes per matrix} = (d_{\text{in}} \cdot r + r \cdot d_{\text{out}}) \cdot b$$

Summing over all the target matrices in each layer and multiplying by the number of layers gives the adapter size. A concrete calculation for Llama-3-70B, rank 16, BF16, all the matrices (Q, K, V, O, gate, up, down):

MatrixDimensionBytes
Q (8192→8192)8192·16 + 16·8192524,288
O (8192→8192)8192·16 + 16·8192524,288
K (8192→1024)8192·16 + 16·1024294,912
V (8192→1024)8192·16 + 16·1024294,912
gate (8192→28,672)8192·16 + 16·28,6721,179,648
up (8192→28,672)8192·16 + 16·28,6721,179,648
down (28,672→8192)28,672·16 + 16·81921,179,648
Sum per layer5,177,344 ≈ 4.94 MB
Adapter total (80 layers)~395 MB

If it is limited to attention-only (Q, O, V, K): about 125 MB per adapter. The choice of which matrices get LoRA belongs to whoever trains it; in serving it is inherited and it determines the cost.

Memory per rank. Linear: rank 8 → ~200 MB; rank 16 → ~400 MB; rank 32 → ~800 MB; rank 64 → ~1.6 GB. The simple sizing rule is: max_lora_rank should be the maximum rank you are going to serve, no more. Setting it higher wastes reserved memory in every slot.

How many adapters fit. For an H100 SXM 80 GB with a Llama-3-70B FP8 base (~70 GB), about 10 GB are left after a minimal KV cache → around 25 fully targeted r=16 adapters or around 80 attention-only. With an AWQ INT4 base (~35 GB), about 45 GB are left → hundreds of adapters. The rule: quantising the base does not just free memory, it multiplies the economics of the platform.

Latency overhead per adapter. Under real reported conditions:

CaseTypical overhead
All requests in the batch on the same adapter~0 % (equivalent to a static merge)
Heterogeneous batch, similar ranks (e.g. all r=16)10-30 % per layer
Heterogeneous batch, mismatched ranks (r=8 with r=128)up to +84 % P95 TTFT for the lower rank
Naive PEFT LoRA (no SGMV)250-950 % extra

It scales linearly with rank: rank 8 ≈ baseline; rank 64 ≈ 3-4 × overhead.

The real implementations in May 2026

ImplementationBase kernelHot-swapQuantised base + LoRAOperational notes
vLLMSGMV + extYes (LoRAResolver, S3/HF/FS plugins)AWQ/GPTQ yes, bnb 4-bit offline onlyThe de facto default. --enable-lora --max-loras N --max-lora-rank R --max-cpu-loras M
LoRAX (Predibase)Optimised SGMVYes (dynamic loading)YesDesigned specifically for multi-LoRA. Supports Medusa adapters per adapter (spec-dec per adapter).
SGLangSGMV / csgmvYesYes--enable-lora-overlap-loading cuts TTFT by up to 78 % on LoRA-heavy workloads
TensorRT-LLMLoRA Executor (C++)Pre-compile at build timeINT4 + LoRA commonPeak throughput on H100/B200, less flexible than vLLM
HF TGIPunica forkYes (LORA_ADAPTERS=...)YesIn maintenance mode as of May 2026; HF recommends vLLM or SGLang
NVIDIA NIMTRT-LLM under the hoodStatic or dynamic (NIM_PEFT_REFRESH_INTERVAL)YesAdapter store per model; polling for hot-add/remove

Three operational observations:

  1. vLLM dominates open-source serving in 2026 thanks to the combination of a mature SGMV + LoRAResolver plugins + support for a quantised base. The critical parameter is --max-lora-rank: many installations set it to the maximum “just in case” and waste memory silently.
  2. LoRAX wins in production operations with thousands of rarely used adapters thanks to its dynamic loading that does not block concurrent requests. A public case: Convirza with 60+ concurrent adapters and sub-2s P95.
  3. SGLang wins on latency when cold starts are frequent thanks to --enable-lora-overlap-loading (async H2D during the compute of the previous request).

Combined pattern with quantisation and disaggregated serving

With quantisation (Quantisation). The canonical stack in May 2026 is base in FP8 (Hopper/Blackwell) or INT4 AWQ + adapters in BF16/FP16. The adapters are not quantised: they are small, the memory saved is irrelevant, and quantisation noise would accumulate badly with the delta. Quantising the base frees massive amounts of memory for more adapters with no significant loss (<1 % in typical MMLU with AWQ INT4, somewhat more in math/code/reasoning). An adapter trained with a BF16 base works with an FP8/INT4 base at inference with marginal loss, and that is what makes QLoRA operationally trivial: train with a 4-bit base and deploy with a consistent 4-bit base.

With disaggregated serving (Disaggregated serving). Multi-LoRA + prefill/decode disaggregation adds a management layer: each pod needs access to the request’s active adapter. The 2026 strategy: replicate hot adapters across every pod (prefill and decode), evict the cold ones. Cold adapters not present in the target pod are transferred on demand, accepting the extra cost in TTFT. Recent work (InfiniLoRA, FASTLIBRA, LoRAServe) automates this balancing, but the simple rule of thumb works in most deployments.

Operational pitfalls

Cold start. The first request to a sleeping adapter involves a fetch (from S3/MinIO/HF Hub) → CPU load → H2D copy. For an adapter of around 400 MB: 0.5-5 s typically, depending on bandwidth. Under high concurrency it can be up to 35 % of E2E latency (Predictive-LoRA paper, arXiv:2512.20210). Established mitigations: SGLang --enable-lora-overlap-loading (cuts TTFT by 35-78 %); vLLM pre-warming with a dummy request when an adapter is registered; predictive prefetching based on patterns (Predictive-LoRA cuts cold start by 68 %).

Heterogeneous worst case. If every request in the batch has a different adapter and a different rank, SGMV loses its advantage because each segment has size 1. Throughput can fall by up to 50 % vs the base without LoRA. Practical mitigation: group adapters by rank in the upstream routing, trying to get requests of the same rank into the same batch step.

Mismatched rank. Co-batching rank 8 with rank 128 penalises the smaller one: +84 % P95 TTFT for the rank-8 requests (Serving Heterogeneous LoRA Adapters paper, arXiv:2511.22880). Practice: normalise the rank across the fleet whenever possible (train every adapter at the same rank, or at least within a narrow range).

Eviction. LRU is the default. If you have more adapters than fit in CPU RAM, the evicted ones get fetched again from cold storage. Monitoring cold_starts_per_minute and cache_hit_ratio per endpoint is basic hygiene.

Base versioning. Every adapter is tied to an exact base version (Llama-3-70B-Instruct ≠ Llama-3.1-70B-Instruct). The routing must validate the base+adapter pair before serving, or the output will be silent rubbish.

Memory fragmentation. Without paged management (the pre-S-LoRA / pre-vLLM case), evicting and inserting adapters fragments HBM until it becomes unusable. Unified Paging solves it: the LoRA weights and the KV cache live in the same pool of blocks, interchangeable.

Implications on on-premise hardware

On an RTX 4090 (24 GB). The classic case: base Llama-3-8B FP8 (~8 GB) or Llama-3-8B AWQ-INT4 (~5 GB) + dozens of r=16 adapters (~80 MB each, around 3 MB per adapter for Llama-3-8B with r=16 attention-only). 50-100 active adapters fit easily on the 4090. It is the natural setup for multi-tenant demos, fine-tuning over an 8B base and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). This is where the serious setup comes in:

  • Llama-3-70B FP8 (~70 GB) fits on 2 H100s; the other 2 GPUs are available for batch + adapters.
  • Llama-3-70B AWQ-INT4 (~35 GB) fits on 1 H100; the rest of the cluster serves more concurrency or more adapters.
  • ~200 fully targeted r=16 adapters fit with a comfortable budget on the cluster, enough for a SaaS platform with dozens of tenants and simultaneous A/B testing.
  • QLoRA training + consistent serving: the train→deploy adapter cycle takes hours, not days, because the adapter is around 400 MB instead of around 140 GB.

The rule of thumb on an H100 cluster in May 2026: an FP8 or INT4 base + 100-500 adapters per cluster is operationally trivial with vLLM or LoRAX; going past a thousand concurrent adapters starts to require serious tuning of eviction and prefetch.

Typical production stack

[API Gateway]
    ↓ (JWT with tenant_id / API key)
[Router]
    ↓ injects adapter_id into the request
[vLLM / LoRAX with --enable-lora]
    --max-loras 16
    --max-lora-rank 32
    --max-cpu-loras 200
    + LoRAResolver → s3://adapters/{tenant_id}/{version}/
    + Base: Llama-3-70B-FP8 loaded once on 4×H100 TP=4
        ↓
    [GPU]
        FP8 base (70 GB) + ~150 BF16 adapters hot in HBM + ~1,000 warm in RAM
[MinIO / S3]
    Cold storage for thousands of adapters
[CI pipeline]
    → trains new QLoRA adapter → push MinIO → notify server → warm-up
[Observability]
    Prom: active_adapters, cache_hit_ratio, cold_starts_per_minute,
          per_adapter_throughput, P99_with_lora vs P99_base

Routing pattern: Authorization: Bearer <key> → middleware extracts tenant_id → maps to adapter_id (Postgres or Redis) → POST /v1/completions with model: "<adapter_id>".

What we have not covered

  • DoRA (Weight-Decomposed LoRA): decomposes the update into magnitude + direction, closing part of the quality gap with full fine-tuning. Supported by TensorRT-LLM and others, but the serving pattern is identical to LoRA.
  • MoE + LoRA: how adapter fine-tuning is done over an MoE, what happens to the routing. Not trivial, an active research area in 2026.
  • Activated LoRA (arXiv:2512.17910): a variant that reuses the KV cache between compatible adapters, cutting the cost of cold start with shared prefixes.
  • LoRA for speculative decoding: each adapter brings its own Medusa head, supported by LoRAX as “Turbo LoRA”.
  • Compress-then-Serve (arXiv:2407.00066): quantising the adapters themselves to serve even more concurrently. Still a marginal practice in production as of May 2026.

See also

References