Slm on lo0 — Tech Blog

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

Tue, 09 Jun 2026 03:00:00 +0000

This is the operational companion to QLoRA and multi-LoRA at the limit on small models. That post takes apart the why, NF4, double quantisation, paged optimizers, the adapter maths; this is the how, with commands you copy and paste. If you have not read the fundamentals one, read it first: here we assume you know what an adapter is, why the base lives in 4-bit and why the gradient only touches the adapter.

TL;DR

A reproducible procedure in five phases: (1) pin the environment with fixed versions; (2) prepare the dataset in chat format; (3) train the QLoRA adapter with TRL + PEFT on an RTX 4090 (24 GB, Ada Lovelace) using gradient checkpointing, gradient accumulation and paged_adamw_8bit; (4) validate and version the adapter as an artefact of megabytes; (5) serve it in vLLM with --enable-lora, loading it hot without restarting the server and resolving it from object storage. All on-premise, on consumer hardware, without a single data item leaving the perimeter. What follows are the exact commands and the memory budget that separates “it fits” from “OOM”.

The end-to-end flow

Phase 0 — Environment and versions

QLoRA is sensitive to the versions of bitsandbytes, transformers, peft and trl: misaligned combinations give dequant errors or adapters that do not load in vLLM. Pin the environment and do not touch it mid-campaign. Reference versions as of June 2026 (check the specific ones in your index; the exact pin matters less than the coherence between them):

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Training (producer)
pip install "torch>=2.4" \
 "transformers>=4.50" \
 "peft>=0.14" \
 "trl>=0.15" \
 "bitsandbytes>=0.45" \
 "accelerate>=1.2" \
 "datasets>=3.2"

# Serving (consumer) — in its own environment/image
pip install "vllm>=0.8"

What each piece does and why it is pinned:

Package	Role in the flow	Why the version matters
`torch`	tensor runtime and CUDA kernels	the CUDA ABI has to match the driver and `bitsandbytes`; a major jump breaks the 4-bit kernels.
`transformers`	loads the base, the tokenizer and the `chat_template`	it has to know the architecture of the SLM you use; a new model needs a version that supports it.
`peft`	implements LoRA/QLoRA: injects the `A,B` matrices and writes the `adapter_config.json`	that `adapter_config.json` is what vLLM reads when serving; old versions write fields the serving side does not understand.
`trl`	the `SFTTrainer`: the supervised training loop	it integrates `peft` natively; its API (`SFTConfig`) changes between versions, hence the pin.
`bitsandbytes`	the NF4 quantisation and `paged_adamw_8bit`	the most sensitive piece: a badly compiled binary gives corrupt dequant or hangs on the first step.
`accelerate`	orchestrates device, mixed precision and `device_map`	the silent backend of almost everything; misaligning it with `transformers` gives cryptic errors.
`datasets`	loads the JSONL (and allows streaming if the corpus is large)	not very sensitive; any recent 3.x will do.
`vllm`	the multi-LoRA serving	separate environment or image: do not mix its stack with the training `bitsandbytes`.

The golden rule: coherence between the four above (transformers, peft, trl, bitsandbytes) matters more than the exact number of each. Pin them when starting a campaign and do not move them until it closes.

Check that the GPU and CUDA are healthy before starting; a badly compiled bitsandbytes shows up late:

python -c "import torch, bitsandbytes; print(torch.cuda.get_device_name(0), torch.cuda.is_available())"
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

For 100 % sovereignty: download the base once from your internal Hugging Face mirror (or a MinIO with the weights) and export HF_HOME to a local volume. Nothing in this flow needs to leave the perimeter.

Phase 1 — Preparing the dataset

The canonical format for a conversational task is JSONL, one conversation per line, with the model’s chat template. Do not invent your own format: use the base tokenizer’s chat_template, because any mismatch between how you train and how you serve degrades quality silently.

{"messages":[{"role":"system","content":"You are a network support assistant."},{"role":"user","content":"The north wing AP does not come up after the outage."},{"role":"assistant","content":"First confirm the PoE on the port..."}]}
{"messages":[{"role":"user","content":"Generate the VLAN change for customer 42."},{"role":"assistant","content":"interface GigabitEthernet0/3\n switchport access vlan 42..."}]}

What each field is and why:

Field	What it is	Operational note
`messages`	the complete conversation, a list of turns	one conversation per JSONL line; it is what `apply_chat_template` turns into tokens.
`role`	who is speaking: `system`, `user`, `assistant`	the adapter learns to produce the `assistant` turns; the `user`/`system` ones are context, not target.
`content`	the text of the turn	the `system` one sets the persona/task; keep it identical to the one you will use in production or the adapter drifts out of alignment.

Operational rules that save grief: watch the ratio of examples (a well-curated narrow-task dataset of 2,000-20,000 examples performs better than 200,000 noisy ones), deduplicate, and set aside 5-10 % as a validation split that does NOT go into training. Building the corpus from production signal is covered by Retrain: closing the loop.

Phase 2 — The training script

A minimal, complete script with TRL + PEFT. It trains an r=8 adapter on an 8B SLM quantised to NF4. Each block has its rationale commented.

# train_qlora.py
import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen3-8B" # or whichever SLM you serve; ALWAYS use the same one in train and serve
OUT = "adapters/soporte-redes-v1"

# 1) Base frozen and quantised to 4-bit NF4 with double quantisation
bnb = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_quant_type="nf4", # NormalFloat, quantile-optimal for Gaussian weights
 bnb_4bit_use_double_quant=True, # quantises the scale constants -> ~0.37 bits/param less
 bnb_4bit_compute_dtype=torch.bfloat16 # the matmuls run in BF16 after dequant on the fly
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
 BASE, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map={"": 0}
)

# 2) The adapter: low rank, attention projections only (aggressive). Raise target_modules if the eval asks for it.
peft_cfg = LoraConfig(
 r=8, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

ds = load_dataset("json", data_files={"train": "data/train.jsonl",
 "eval": "data/eval.jsonl"})

# 3) Training config designed to fit in 24 GB
cfg = SFTConfig(
 output_dir=OUT,
 per_device_train_batch_size=1, # small real batch
 gradient_accumulation_steps=16, # EFFECTIVE batch = 1*16 = 16, without paying its VRAM all at once
 gradient_checkpointing=True, # recomputes activations in backward: trades compute for memory
 optim="paged_adamw_8bit", # paged optimizer: the airbag against VRAM peaks
 learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
 num_train_epochs=3, bf16=True,
 max_length=2048, # bounds the sequence: activations scale with it
 logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=200,
 report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
 train_dataset=ds["train"], eval_dataset=ds["eval"],
 processing_class=tok)
trainer.train()
trainer.save_model(OUT) # saves ONLY the adapter (MB), not the base

`BitsAndBytesConfig` — how the base is quantised

Option	What it does	Why this value / when to change it
`load_in_4bit=True`	loads the base weights in 4-bit	it is the foundation of QLoRA: without it the 8B does not even fit to train.
`bnb_4bit_quant_type="nf4"`	uses the NF4 format (quantile-optimal for Gaussian weights)	`"fp4"` exists, but NF4 performs better on transformer weights; leave NF4.
`bnb_4bit_use_double_quant=True`	quantises the scale constants themselves	saves ~0.37 bits/param (hundreds of MB on an 8B); the margin separating “it fits” from “OOM”. Leave it at `True`.
`bnb_4bit_compute_dtype=torch.bfloat16`	matmul precision after undoing the quantisation on the fly	BF16 on Ada/Hopper (4090, H100); use `float16` only on GPUs without BF16.

`LoraConfig` — the shape of the adapter

Option	What it does	Why this value / when to change it
`r=8`	adapter rank: its correction capacity	4-8 for a narrow task (aggressive); raise it to 16-64 only if the eval shows underfitting.
`lora_alpha=16`	scale factor of the delta (effective `α/r`)	common convention `α=2r`; it modulates how much the adapter “weighs” on the base.
`lora_dropout=0.05`	regularisation on the adapter	0.05-0.1 with small datasets (avoids overfit); 0 if the corpus is large.
`bias="none"`	does not train the bias terms	`"none"` is the standard; `"all"`/`"lora_only"` rarely help and cost params.
`task_type="CAUSAL_LM"`	objective/head type	fixed for a generative LLM.
`target_modules=[q,k,v,o]`	which matrices get an adapter	attention only = cheap and aggressive; add `gate_proj`/`up_proj`/`down_proj` (MLP) if the task demands rewriting more behaviour and the eval asks for it.

`SFTConfig` — the memory budget and the loop

Option	What it does	Why this value / when to change it
`per_device_train_batch_size=1`	microbatch per GPU	1 on 24 GB; the real batch is built by `gradient_accumulation_steps`.
`gradient_accumulation_steps=16`	accumulates 16 microbatches before updating	effective batch = 1×16 = 16 without paying its VRAM all at once; raise it if you shorten the sequence and want a bigger effective batch.
`gradient_checkpointing=True`	recomputes activations in the backward pass instead of storing them	essential on a 4090: ~20-30 % slower in exchange for much less VRAM.
`optim="paged_adamw_8bit"`	Adam optimizer in 8-bit + states pageable to RAM	less state VRAM and the airbag that avoids the OOM at the peaks.
`learning_rate=2e-4`	learning rate of the adapter	1e-4-3e-4 is the typical QLoRA range; adapters tolerate a higher LR than a full fine-tune.
`lr_scheduler_type="cosine"`	LR decay curve	`cosine` or `linear`; cosine usually gives a smooth drop at the end.
`warmup_ratio=0.03`	warms the LR over the first 3 % of steps	avoids the instability of the first steps.
`num_train_epochs=3`	complete passes over the dataset	1-3; watch the eval loss so as not to overfit.
`bf16=True`	compute and adapter precision	BF16 on Ada/Hopper; `fp16=True` if your GPU has no BF16.
`max_length=2048`	maximum sequence length	the #1 VRAM lever for activations: shorten it first if there is an OOM.
`eval_strategy`/`eval_steps`/`save_steps`	validation and checkpoint cadence	tune them to the dataset size; evaluating often costs time.

The four pieces that make it fit on a 4090 are: per_device_train_batch_size=1 + gradient_accumulation_steps (a large effective batch without its memory cost all at once), gradient_checkpointing=True (recomputing activations instead of storing them) and optim="paged_adamw_8bit" (paging states to RAM at the peaks). Remove any of the three with long sequences and you will see the OOM.

A declarative alternative with Axolotl if you prefer YAML over Python (same result):

base_model: Qwen/Qwen3-8B
load_in_4bit: true
adapter: qlora
lora_r: 8
lora_alpha: 16
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
sequence_len: 2048
micro_batch_size: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
optimizer: paged_adamw_8bit
learning_rate: 0.0002
num_epochs: 3
bf16: true
datasets:
 - path: data/train.jsonl
 type: chat_template

Phase 3 — Launch and monitor

# Simple launch on one GPU
python train_qlora.py

# In another terminal: watch VRAM. If it gets close to the ceiling, lower max_length or raise grad accumulation.
watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Approximate VRAM budget when training the 8B on the 4090, and what to touch when it gets tight:

Component	Approx. VRAM	Lever if there is an OOM
8B NF4 base (frozen)	~4.0 GB	— (fixed)
Adapter + grad + Adam states	~0.3-0.7 GB	lower `r`
Activations (batch × sequence)	~6-14 GB	lower `max_length`, `batch_size`; raise `grad_accum`
Dequant buffers / workspace	~1-2 GB	—

Table of quick OOM remedies, in order of cost: (1) lower max_length; (2) confirm gradient_checkpointing=True; (3) raise gradient_accumulation_steps and lower per_device_train_batch_size to 1; (4) use paged_adamw_8bit (already in the script); (5) as a last resort lower r. If after all that it still does not fit, the sequence or the model is too large for 24 GB: either you bound it, or you move up the hardware.

Phase 4 — Validating the adapter

Never promote an adapter on the strength of the training loss. Measure against the validation split you set aside and against a handful of real prompts.

# quick_eval.py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
 bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, "adapters/soporte-redes-v1") # base + adapter

msgs = [{"role": "user", "content": "The north wing AP does not come up after the outage."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(0)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0], skip_special_tokens=True))

For a serious verdict, run the adapter through your eval suite (the layer described by LLM evals) and compare against the base without an adapter and against the previous version of the adapter. Promote only if it wins on the task metric without regressing on safety/format.

Phase 5 — Versioning the adapter as an artefact

The adapter is a pair of files of a few MB (adapter_model.safetensors + adapter_config.json). Treat it as a versioned, signed, traceable artefact, not as a loose file.

# Reproducible checksum + upload to internal object storage (MinIO/S3)
sha256sum adapters/soporte-redes-v1/adapter_model.safetensors > adapters/soporte-redes-v1/SHA256
aws --endpoint-url https://minio.interno s3 cp \
 adapters/soporte-redes-v1/ s3://adapters/soporte-redes/v1/ --recursive

A convention that works: s3://adapters/<task-or-customer>/<version>/. Immutable per version, with its SHA256. Deleting a customer is deleting a prefix of a few MB, not retraining anything. Versioning 500 adapters costs what versioning 500 heavy configuration files costs.

Phase 6 — Serving in multi-LoRA with vLLM

The consumer loads one shared base and applies the adapter delta per request. Startup with static adapters declared:

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3-8B \
 --enable-lora \
 --max-loras 8 \  # max nº of DIFFERENT adapters per batch (not the total loadable)
 --max-lora-rank 8 \  # = the maximum rank of your adapters; do not inflate it (wastes memory)
 --max-cpu-loras 64 \  # adapters cached in RAM for fast swap to VRAM
 --lora-modules soporte-redes=/srv/adapters/soporte-redes/v1

Each flag, what it controls and how to size it:

Flag / variable	What it controls	How to size it
`--enable-lora`	turns on adapter support	mandatory; without it, vLLM ignores any `model` that is an adapter.
`--max-loras 8`	nº of different adapters in a single batch	more adapters per batch makes the SGMV kernels more expensive; 8-32 is reasonable. It is not the total loadable.
`--max-lora-rank 8`	maximum rank the server reserves	set it equal to the real rank of your adapters (8 here); inflating it wastes VRAM and performance.
`--max-cpu-loras 64`	adapters cached in RAM ready to page to VRAM	≥ nº of active adapters; it is the “bench” from which fast swapping happens.
`--lora-modules name=path`	declares static adapters at startup	useful for the fixed ones; leave it out if everything goes through dynamic loading/Resolver.
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`	enables the hot load/unload endpoints	essential for `/v1/load_lora_adapter`; without it, the server is static.

--max-loras limits the different adapters per batch, not how many you can have loaded; the bulk lives on CPU (--max-cpu-loras) and is paged to VRAM on demand. Set --max-lora-rank to the real rank (8 here): inflating it wastes memory and performance. Requests pick the adapter through the model field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "soporte-redes",
 "messages": [{"role":"user","content":"The north wing AP does not come up after the outage."}]
}'
# model:"Qwen/Qwen3-8B" (no adapter) uses the bare base on the same server

Hot loading of a new adapter without restarting (thanks to VLLM_ALLOW_RUNTIME_LORA_UPDATING=True):

curl -X POST http://localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" -d '{
 "lora_name": "cliente-42",
 "lora_path": "/srv/adapters/cliente-42/v3"
}'
# and to free VRAM/CPU when a customer goes idle:
curl -X POST http://localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" -d '{"lora_name":"cliente-42"}'

For multi-tenant at scale, avoid declaring hundreds of adapters by hand: the LoRAResolver resolves and loads the adapter from local storage or S3 the first time an unknown model arrives, so the server stays lean and the adapters are pulled lazily from your MinIO. The internals of how thousands of concurrent adapters are batched (SGMV kernels, unified paging, the heterogeneous gather/scatter) are in Multi-LoRA serving; this runbook only switches them on. To squeeze the base’s decode throughput on a 4090, combine this with Optimising decode in vLLM.

Serving multi-adapter vs merging per task

Two deployment architectures, and the procedure changes:

Serving multi-LoRA (the above). A shared base + N hot adapters. It is the default sovereign pattern: minimum footprint, per-customer isolation, hot-swap. Use standard QLoRA and merge nothing.

Merging per task. If you want a single quantised-and-adapted artefact per task (no adapter at runtime), do not merge a standard QLoRA adapter into the 4-bit base: the merge reintroduces precision that NF4 does not represent and on requantising you lose part of what was learnt. For that case, train with QA-LoRA (quantization-aware), which merges cleanly onto a quantised base. It is an architecture decision, not a quality one; the conceptual detail is in the fundamentals post.

Checklist of operational gotchas

Coherent chat template between training and serving. The most common and most silent mismatch: you train with one chat_template and serve with another. Use the base’s on both sides.
Exactly the same base (revision included) in train and serve. An adapter trained on Qwen3-8B is not valid on another revision of the model.
--max-lora-rank ≥ the rank of ALL the adapters served together, but no more: inflating it wastes VRAM.
KV budget vs --max-loras. The bottleneck in serving is not the adapters (MB), it is the KV cache and concurrency; see Inverted roofline for the SLM regime.
r too low = underfitting if the task demands rewriting a lot of behaviour. Raise r only if the eval asks for it.
Do not promote on training loss. Validate against the reserved split + real prompts + a safety regression.
Version and make immutable every adapter with its SHA256; never overwrite a served version.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB) the same machine is producer and consumer: you train a customer’s adapter in hours and serve it on the same server on top of the shared base. It is the canonical case for multi-tenant demos and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8) QLoRA stops being necessary in order to fit, but it serves to parallelise production (several adapter jobs at once) and to keep the quantised format consistent between training and serious serving of hundreds of concurrent adapters. The base can run in native FP8; the mechanics of the runbook do not change, only the scale.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hugging Face TRL — PEFT integration (SFTTrainer + QLoRA): https://huggingface.co/docs/trl/peft_integration
Hugging Face PEFT: https://github.com/huggingface/peft
bitsandbytes: https://github.com/bitsandbytes-foundation/bitsandbytes
vLLM — LoRA Adapters (serving, dynamic loading, LoRAResolver): https://docs.vllm.ai/en/stable/features/lora/
Axolotl: https://github.com/axolotl-ai-cloud/axolotl
Xu, Y. et al. QA-LoRA: Quantization-Aware Low-Rank Adaptation. ICLR 2024. https://arxiv.org/abs/2309.14717

QLoRA and multi-LoRA at the limit on small models

Tue, 09 Jun 2026 02:30:00 +0000

This post is the training-side companion to Multi-LoRA serving. That one takes apart the consumer, how hundreds of concurrent adapters are served with SGMV kernels and unified paging; this one takes apart the producer, how an adapter is trained on top of a quantised base on a single GPU, and why the pattern “one frozen base SLM + N low-rank adapters” is the natural fit for small models. We do not repeat the serving internals here; we assume you have read them.

TL;DR

QLoRA (Dettmers et al., NeurIPS 2023) solves a specific problem: fine-tuning a model without having the VRAM to hold its weights in BF16, its gradients and the optimiser states. The idea has three pieces. One: freeze the base and quantise it to 4-bit with a new format, NF4 (NormalFloat 4-bit), quantile-optimal for weights distributed almost like a Gaussian. Two: do not train the base, not a single one of its weights moves, but rather a pair of small LoRA matrices in BF16 plugged in parallel; the gradient flows only through that adapter. Three: two memory tricks, double quantisation (quantising the quantisation constants themselves) and paged optimizers (optimiser states that page out to RAM when VRAM gets tight). The measurable operational result: a 3-8B SLM is fine-tuned on an RTX 4090 (24 GB, Ada Lovelace), not on a cluster. And because the output of training is an adapter measured in megabytes, not gigabytes, the pattern that emerges is a single frozen base SLM in 4-bit plus N adapters, one per customer, domain or task, served on top of the shared base with the stack we already covered in multi-LoRA serving. Per-customer isolation, minimum footprint, sovereign deployment.

The analogy: the frozen guitar and the swappable pedalboard

Think of a session guitarist recording for very different clients: a jazz record, a corporate jingle, a metal track. He has one single guitar, his trusted instrument, set up, with a base tone he knows by heart. What he does not do is buy a new guitar for every song. What he does is keep a pedalboard: a distortion pedal, a chorus, a delay. For each track he plugs in the pedal that fits, and the same guitar sounds completely different.

The mapping is exact:

The guitar = the base SLM. A single copy, set up at the factory, frozen. In QLoRA it is also kept in a compressed case: quantised to 4-bit. You do not touch it: you do not change its pickups or adjust the neck. It weighs what it weighs and there it stays.
Each pedal = a LoRA adapter. Small, cheap, specific to one sound. You train it for a task and put it in a drawer.
Training QLoRA = designing a new pedal by listening to the (frozen) guitar through it, adjusting only the pedal’s pots until it sounds the way you want. The guitar’s base tone is not modified; you learn the correction the pedal applies on top.
Serving multi-LoRA (Multi-LoRA serving) = having the whole pedalboard set up on stage and choosing the right pedal per note, per request. The guitar is the same; what changes between requests is which pedal is active.

The analogy holds down to the detail that confuses people most: the training gradient only “touches” the pedal. The guitar is frozen in its compressed case; learning does not move it. That is what lets the base live in 4-bit throughout the fine-tuning without the quantisation getting in the way: no gradient is ever computed for it.

The bare mechanism: LoRA, and why you can train on a 4-bit base

A minimal reminder of LoRA (Hu et al., ICLR 2022). An adapter modifies a matrix W of the base by adding a low-rank product to it:

$$W' = W + B A, \qquad A \in \mathbb{R}^{r \times d}, \quad B \in \mathbb{R}^{d \times r}$$

with r the rank, much smaller than d. In the forward pass BA is not materialised; what is computed is:

$$y = W x + B(A x)$$

The base compute (Wx) happens just the same; the adapter adds two cheap matmuls. The key to QLoRA is who receives a gradient. The base W is frozen: ∂L/∂W is neither computed nor stored. Only A and B are trainable. That is why W can live quantised to 4-bit without trouble: in the forward pass the quantisation is undone on the fly to compute Wx (dequant → matmul in BF16), but since W is never updated it does not need the precision of a trainable weight. The adapter A, B is in BF16, and it is the only path the gradient flows through.

This is what breaks the memory wall. In a full fine-tuning you need, per weight: the weight (2 bytes BF16), its gradient (2 bytes), and the two Adam states (momentum and variance, typically 4+4 bytes in FP32), on the order of 12-16 bytes per trainable parameter. With QLoRA, the base weights take 0.5 bytes (4-bit) and have no gradient and no optimiser states. Only the few million adapter parameters pay the 16-byte cost. For an 8B, that is the difference between ~130 GB and fitting in 24 GB.

NF4: why a new format instead of INT4

QLoRA does not use linear INT4 for the base, but NF4 (NormalFloat 4-bit). The intuition: the weights of a trained transformer are distributed, empirically, very close to a zero-centred Gaussian. INT4 spreads its 16 levels uniformly across the range, which wastes levels in the tails (where there are almost no weights) and leaves few in the centre (where they pile up). NF4 spreads the 16 levels according to the quantiles of a normal distribution: more levels where there is more probability mass. It is, by construction, information-theoretically optimal for exactly Gaussian data, each level covers roughly the same number of weights. It is also symmetric about zero and guarantees an exact representation of 0 (important for sparsity and padding). The detail of the quantisation formats is in Quantisation for LLM inference; here the idea that NF4 spends its bits where the weights are is enough.

Double quantisation and paged optimizers

Quantising to 4-bit is not entirely free: for each block of weights (typically 64) you need to store a scale constant in FP32 so the quantisation can be undone. Those constants weigh something. With blocks of 64 and one FP32 scale (32 bits) per block, that is 32/64 = 0.5 bits per parameter in metadata alone, a 12.5 % overhead on top of the 4 useful bits. Double quantisation attacks that: it quantises the scale constants themselves (to 8-bit, in blocks of 256), bringing the overhead down to ~0.127 bits/param. Quantising the quantisation sounds recursive and it is; the saving is small in absolute terms (~0.37 bits/param) but on an 8B that is hundreds of MB, which is exactly the margin separating “it fits” from “it does not fit” on a 4090.

Paged optimizers attack the memory peaks. During training, certain moments, a batch with a very long sequence, a large activation, push VRAM close to the limit and blow up with an OOM. The idea, borrowed from operating system paging, is to allocate the optimiser states in NVIDIA unified memory: when VRAM gets tight, those pages are evicted to host RAM automatically and brought back when needed. It speeds nothing up; it avoids the crash at the peaks. It turns an “intermittent OOM” into “a bit slower at the worst moments”, which for an unattended training run on a single GPU is the difference between finishing and not finishing.

“Aggressive” training: very low rank and QA-LoRA

“Aggressive” in this context means two things, sometimes combined.

Very low rank (r = 4-8). The rank is the bottleneck of the correction: how much “capacity” the adapter has to steer the base away from itself. A high rank (64, 128) brings the adapter closer to a full fine-tuning but weighs more and takes longer to train. For an SLM adapted to a narrow, well-defined task, an output format, a lexical domain, a response style, a rank of 4-8 is usually enough, and the resulting adapter weighs a fraction. The risk of a low rank is underfitting: if the task demands rewriting a lot of the base’s behaviour, r=4 falls short. The honest rule is empirical: raise the rank only if the eval asks for it, not “just in case”. On small SLMs, where the base has less spare capacity, a low rank tends to work proportionally better than on large models, but this depends on the task and has to be measured, not assumed.

QA-LoRA (quantization-aware LoRA, Xu et al., arXiv:2309.14717). There is a subtle friction in standard QLoRA: you train the adapter in BF16 against a 4-bit base, but if you then want to merge the adapter into the base (W' = W + BA) to serve a clean quantised model, the merge reintroduces precision that the 4-bit format cannot represent, and on requantising you lose part of what was learnt. QA-LoRA trains the adapter aware of the target’s quantisation: it balances the degrees of freedom of the quantisation and of the adaptation (with group-wise quantisation) so that, when it finishes, the adapter merges cleanly into a quantised base without a requantisation step that degrades it. The result is a final model that is quantised-plus-adapted, with no separate adapter at runtime, useful when you want a single deployable artefact per task instead of the shared-base + adapters pattern. The choice between “QLoRA + serve multi-adapter” and “QA-LoRA + merge per task” is a deployment architecture decision, not one of pure quality.

The maths that matters

Three calculations drive any QLoRA decision on SLMs.

Adapter parameters. For each target matrix of dimension d with rank r, the adapter contributes A (r×d) plus B (d×r), that is 2·r·d parameters. Summing over the target matrices and multiplying by the number of layers:

$$\text{params}_{\text{adapter}} = L \cdot \sum_{\text{matrices}} 2 \cdot r \cdot d$$

Worked example — Llama-3-8B, attention (q, k, v, o), d = 4096, L = 32 layers, r = 8. Taking the four attention projections with the same d = 4096 (a simplification; in Llama-3 K and V are narrower because of GQA, which gives even fewer params):

$$\text{params} \approx 32 \cdot 4 \cdot (2 \cdot 8 \cdot 4096) = 32 \cdot 4 \cdot 65\,536 \approx 8.4\text{M params}$$

In BF16 (2 bytes/param): 8.4M · 2 ≈ 16.8 MB ≈ ~17 MB. Seventeen megabytes. Compare that with the base: an 8B in NF4 takes 8\text{G} · 0.5\,\text{bytes} ≈ 4\text{ GB} (plus the small constants overhead after double quantisation). The adapter is 0.4 % of the size of the quantised base. This is what makes having hundreds operationally trivial: an adapter is not a model, it is almost a heavy configuration file.

How many adapters fit on a 4090 after the base + KV? Budget for an RTX 4090 (24 GB): 8B NF4 base ~4 GB, leave ~5 GB for KV cache and inference activations at moderate concurrency → that leaves ~15 GB free (being conservative, call it ~12-15 GB). With ~17 MB adapters (r=8, attention-only):

$$\frac{15\,000\ \text{MB}}{17\ \text{MB/adapter}} \approx 880 \text{ adapters}$$

On the order of thousands if you cut the reserved KV cache or use rank 4 (~8.5 MB/adapter → ~1750 in 15 GB). The bottleneck is never the adapters’ space; it is the KV cache and concurrency. For the details of how those thousands are served concurrently, the heterogeneous batching, the unified paging, the SGMV kernels, see Multi-LoRA serving. The relevant summary here: the adapter compute is nearly free (low rank, two thin matmuls); the serving performance challenge is not that compute but the gather/scatter of the right adapters per batch row when a single batch mixes requests from different adapters. That is the consumer’s problem, not the producer’s.

QLoRA training VRAM in 24 GB. The rough budget to fine-tune the 8B on a 4090:

Component	Approx. VRAM
8B base in NF4 (frozen weights)	~4.0 GB
Adapter (BF16 params + gradient + Adam states, ~16 B/param over ~8-40M params)	~0.3-0.7 GB
Activations (depends on batch and sequence length; the variable bulk)	~6-14 GB
Dequant buffers, scales, workspace	~1-2 GB
Total	fits in 24 GB with margin

The large, variable piece is the activations, which scale with batch × sequence length. That is why real QLoRA on a 4090 is done with a small batch + gradient accumulation (simulating a large batch by accumulating gradients over microbatches) + gradient checkpointing (recomputing activations in the backward pass instead of storing them, trading compute for memory) + bounded sequences. The paged optimizers are the airbag for the activation peaks that would otherwise blow up. The claim “QLoRA fine-tunes an 8B on a 4090” is true with that configuration; with long sequences, a large batch or a high rank, it does not fit. As with any number, the methodology matters more than the headline.

The fit with small models and sovereignty

This is where QLoRA + SLM stops being a VRAM trick and becomes an architectural pattern.

An SLM (3-8B) already fits comfortably on a single GPU for inference. If on top of that the base lives in 4-bit (~4 GB for an 8B), you have memory to spare. What QLoRA enables is for that same machine, the 4090, to be both the producer and the consumer: you train a new customer’s adapter in hours, on the same class of hardware where you then serve it. The artefact that travels between “train” and “deploy” is an adapter of MB, not GB: it is versioned, signed, moved across the network, stored in MinIO/S3 without thinking about the cost.

The sovereign pattern falls out on its own:

Per-customer isolation. Each customer has their adapter, trained only on their data. The base is generic and shared; what is specific to the customer lives isolated in their (A, B) pair. Deleting a customer is deleting a file of a few MB, not retraining anything.
Minimum footprint. One base + N adapters fits where N bases would not come close. The economics of “one model per customer” (tens of GB each) are prohibitive; those of “one base + adapters” (MB each) are trivial. It is exactly the difference between the pedalboard and buying a guitar per song.
Sovereign deployment. Everything fits on-premise, on your hardware, without a single data item leaving the perimeter. Training (QLoRA on the 4090) and serving (multi-LoRA on the same base) live inside. There is no dependency on an external API for fine-tuning or for serving.

The choice between adapting by domain (one adapter per knowledge area) and retrieving by context (RAG that injects the knowledge into the prompt) is real and not mutually exclusive: the adapter changes the model’s behaviour and style, RAG changes the facts it has access to. The sibling post on aggressive RAG on small models in this series works through it; the short rule is: adapt what is stable and behavioural, retrieve what is volatile and factual.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

This is QLoRA’s natural workbench. The canonical case: a 3-8B base SLM in NF4, fine-tuning an r=8 attention-only adapter, with gradient checkpointing + gradient accumulation + paged optimizer. It trains in hours for narrow-task datasets (thousands to tens of thousands of examples), and the same machine then serves the base + tens or hundreds of adapters for multi-tenant demos and platform prototypes. The 4090 is where QLoRA went from “a paper technique” to “anyone with a consumer GPU can do it”, and that is exactly its value. The honest rule: it fits with the memory configuration described; with long sequences, a large batch or a high rank, move up the hardware.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here QLoRA stops being strictly necessary in order to fit, an 8B in BF16 has room to spare, but it remains useful for another reason: parallelising adapter production. With 320 GB and native FP8 you can train several adapters at once (one job per customer, several in parallel), or fine-tune somewhat larger models with QLoRA without TP. The consumer on this cluster is the serious setup from Multi-LoRA serving: an FP8 base + hundreds of concurrent adapters. The rule of thumb: on the 4090, QLoRA is the tool to be able to fine-tune; on the H100 cluster, it is the tool to fine-tune many at once, cheaply, keeping the quantised format consistent between training and serving.

What we have not covered

The internals of heterogeneous serving (SGMV kernels, MBGMM/MBGMV, unified paging, cold start, eviction): they are covered in full in Multi-LoRA serving. This post is deliberately the producer’s side.
DoRA and variants (magnitude-direction decomposition): they close part of the gap with full fine-tuning; different training pattern, identical serving pattern.
Sub-4-bit and ternary quantisation of the base: what happens when the base drops from NF4 to 2-bit or ternary underneath the adapter; the sibling post in the series works through it.
Collecting the fine-tuning dataset: how each adapter’s corpus is built from production feedback is in Retrain: closing the loop.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. ICLR 2024. https://arxiv.org/abs/2309.14717
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Official QLoRA / bitsandbytes repo: https://github.com/artidoro/qlora
Hugging Face PEFT (LoRA, QLoRA): https://github.com/huggingface/peft

Aggressive RAG in small models: trading parameters for retrieval

Tue, 09 Jun 2026 02:20:00 +0000

This post belongs to the series on inference performance in small models. Its sibling piece, The roofline inverts in small models, explains why the compute-bound prefill is the bottleneck that shapes the whole discussion here. It is worth reading first: here we assume that adding more context is not free.

TL;DR

An SLM (say 1B–8B parameters) knows fewer facts than a 70B–700B model, simply because it has fewer weights in which to memorise them. But its ability to reason over text in front of it, following instructions, extracting, synthesising, comparing, degrades far less with size than its encyclopaedic knowledge does. The operational consequence is direct: use the SLM as a reasoning engine over curated context, not as a database. Move knowledge from the weights into the context via retrieval. The problem is that “aggressive retrieval” is often read as “put in lots of chunks”, and that collides head-on with three facts about SLMs: shorter context windows, worse use of long context (the lost in the middle effect is more severe the smaller the model) and a compute-bound prefill whose cost grows with the context length $C$, linearly in the projections and quadratically in attention. You cannot simply add tokens. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb either cost. The way out is not to retrieve less, but to retrieve better: reranking for precision over recall, compressing the context before injecting it, prefix caching the stable documents, semantic caching of answers, and structured output with external tools that stand in for internal knowledge. This post works through the maths and gives a TTFT number before and after compressing a context from 4,000 to 1,000 tokens on an RTX 4090.

The analogy: the open-book exam

Two students sit the same exam. The first has a prodigious memory: they have memorised the entire syllabus, paragraph by paragraph. The second has an ordinary memory, forgetting dates, confusing names, but is allowed to bring in a crib sheet.

If the second student’s crib sheet is a chaos of piled-up photocopies, they lose: they take too long to find what they are after, get distracted by irrelevant pages and run out of time. But if their crib sheet is excellent, trimmed to the essentials, reordered by relevance, with the important things at the top and no filler, then they not only do not lose: they often win, because they reason just as well as the first student and, on top of that, work from verified material instead of from blurry recollections they may be inventing.

The moral has three layers, and each maps to an engineering decision:

Memorising everything is expensive. The first student invested months. A large model invests parameters, and VRAM, and inference FLOPs, in memorising facts.
The crib sheet matters more than its size. A well-made one-page crib sheet beats ten badly organised pages. More retrieved context is not better context: the material’s precision beats volume.
Knowing how to search and synthesise is a different skill from knowing. It is the one the SLM retains. The whole strategy consists of leaning on that skill and subcontracting the memory.

The rest of the post is, essentially, how to build an excellent crib sheet under the constraint that the student (the SLM) reads slowly and gets tired with long texts.

The capacity argument: how many facts fit in the weights

Let us start by justifying the thesis with an order of magnitude, not with faith. How much factual knowledge really fits in a model’s weights?

There is a recurring empirical estimate in the interpretability and memorisation literature: a dense model can store on the order of 2 bits of memorised information per parameter before saturating (the exact figure varies by study and training regime; take it as an order of magnitude, not as a law). An 8B-parameter model then has an information storage ceiling on the order of:

$$8 \times 10^9 \text{ params} \times 2 \text{ bits/param} = 1.6 \times 10^{10} \text{ bits} \approx 2 \text{ GB of information}$$

And that budget is not only for facts: the vast majority goes on grammar, syntax, reasoning ability, code, formatting, and only a fraction is left for encyclopaedic knowledge. Compare it with the other side: a retrievable corpus of several million documents, a corporate wiki, a document repository, a technical knowledge base, easily takes up hundreds of GB to terabytes of text, indexed and queryable with millisecond latency. The asymmetry is two or three orders of magnitude in favour of the external corpus.

The conclusion is not that the weights are useless, they are where reasoning lives, which is the expensive thing to replicate, but that competing with an external index on fact capacity is losing by construction. A 70B model has ~9× more memorisation budget than an 8B one, but it is still negligible against the corpus. That is why the large model also does RAG in production. The difference is that the SLM needs it: without retrieval, its factual knowledge is too sparse and, worse, prone to hallucinating exactly in the gaps it did not memorise.

The central tension: retrieving more is not putting in more

This is where most naive designs break. “Aggressive retrieval” sounds like a large top-k: if retrieval helps, retrieve 20 chunks instead of 5. But in an SLM that fails for two independent reasons, one of quality and one of cost.

(a) SLMs make worse use of long context

The lost in the middle effect (Liu et al., 2023) is well known: LLMs retrieve information placed at the beginning and end of the context better, and the information in the middle worse. What gets less emphasis is that the effect is more severe the smaller the model. An SLM has fewer attention heads, fewer layers and poorer internal representations for “tracking” a relevant fact buried at position 14 out of 20 chunks. Its nominal context window also tends to be shorter (4K–32K against the 128K+ of the large ones), and the effective window, the length beyond which quality collapses, is smaller still. Putting in 20 chunks does not mean the model reads all 20: it means it will probably ignore or misread the ones in the middle, while paying the cost of all of them.

(b) Prefill grows with the context and is compute-bound

This is the blow people underestimate. Prefill, processing the full prompt before emitting the first token, is the compute-bound phase of inference (unlike decode, which is memory-bound; the detail lives in The roofline inverts). Its cost grows with the context length $C$, and it determines TTFT (time to first token). More chunks → more prefill tokens → more TTFT and more compute cost per request. In an SLM, where prefill is proportionally more expensive relative to the model, this hurts especially.

The operational conclusion is uncomfortable but clear: you cannot compensate for fewer parameters simply by putting in more context. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb both costs. The way out is to retrieve less but better, and to compress what you retrieve.

The maths of prefill

Let us put numbers on “prefill grows with the context”. For a context of $C$ tokens, a transformer layer does two classes of work:

Linear projections (QKV, attention output, FFN): each token is multiplied by fixed-size weight matrices. The cost is $O(C)$ in FLOPs, linear in the number of tokens.
Attention ($QK^\top$ and the multiplication by $V$): each token attends to all the others. The cost is $O(C^2)$, quadratic in the number of tokens.

The total prefill cost per layer has the form:

$$\text{FLOPs}_{\text{prefill}} \approx \underbrace{a \cdot C}_{\text{projections}} + \underbrace{b \cdot C^2}_{\text{attention}}$$

with $a$ and $b$ constants that depend on the model dimension. For moderate contexts (a few thousand tokens) in an SLM, the linear term still dominates or is comparable to the quadratic one; the quadratic term becomes dominant at long contexts. What matters: if you compress the context $C \to C/k$, the linear term falls $\times k$ and the quadratic one falls $\times k^2$. Compression is the only lever that attacks both terms at once, and it attacks the worse one disproportionately.

Numerical example: TTFT before and after compressing, RTX 4090

Let us model TTFT as the time to process the prefill tokens at a given prefill throughput. Take an RTX 4090 (24 GB, Ada Lovelace) serving a quantised SLM, with a prefill throughput of ~5,000 tok/s (an illustrative figure; the real value depends on the model, the quantisation and the batch, so measure it, do not assume it).

Let there be a retrieved context of 4,000 tokens (8 chunks of ~500 tokens). Approximating TTFT as dominated by the context prefill:

$$\text{TTFT}_{\text{before}} \approx \frac{4000 \text{ tok}}{5000 \text{ tok/s}} = 0.80 \text{ s}$$

Now we compress that context to 1,000 tokens ($k = 4$). Prefill throughput is not constant with $C$, it drops a little at long contexts because of the quadratic term, but taking the conservative linear tokens/throughput approximation:

$$\text{TTFT}_{\text{after}} \approx \frac{1000 \text{ tok}}{5000 \text{ tok/s}} = 0.20 \text{ s}$$

TTFT falls from 0.80 s to 0.20 s, a $4\times$ reduction in the linear part. But the FLOPs arithmetic is even more favourable in the attention component: that part of the work falls $\sim k^2 = 16\times$. In practice total TTFT does not fall 16× because the cost is not purely quadratic at this scale, but the real reduction lies between 4× and something larger depending on how much attention weighed, and the aggregate compute saving (what pays the electricity bill and frees the GPU for another request) is substantially bigger than the plain 4× of the token count.

The argument generalises: compressing the context by a factor $k$ reduces TTFT by at least $\sim k\times$ and the attention cost by $\sim k^2\times$. For an SLM, where TTFT is often the SLA that matters, this is the difference between an assistant that answers instantly and one that feels slow.

The five levers to resolve the tension

The strategy is not “retrieve less and settle”. It is retrieve aggressively from the index and then distil aggressively what you retrieved before it reaches the SLM. Five levers, in order of application within the pipeline.

1. Aggressive reranking: precision over recall

The initial retriever (dense, sparse or hybrid) optimises recall: it brings back 50–100 candidates so as not to leave anything out. The reranker, a cross-encoder that sees the query and the document together, optimises precision: it reorders those candidates and you keep the 3–5 best. For an SLM this is not a luxury, it is structural: since the model uses long context badly, every chunk that goes in must earn its place. Better 4 chunks of very high relevance than 15 mediocre ones. The detail of hybrid retrieval and reranking is in Reranking and hybrid retrieval; here the rule is enough: maximise recall in the retriever, maximise precision in the reranker, and inject few.

2. Context compression: distilling the crib sheet

Once you have the best chunks, they still contain filler: padding sentences, redundancy, context irrelevant to the specific query. Context compression trims them before injection:

Extractive compression (LLMLingua / LongLLMLingua style, Jiang et al. 2023): a small model scores the perplexity or relevance of each token/sentence with respect to the query and removes the low-information ones, keeping the densest extractive subset. It reduces tokens without a second large generative model in the loop. LongLLMLingua adds position-aware reordering to mitigate lost in the middle.
Abstractive compression: a model summarises the retrieved chunks into shorter text. More aggressive in token reduction, but it introduces a generative step (cost and possible loss of fidelity).
Soft prompts / context distillation: compressing the retrieved context not into text, but into a handful of embeddings/soft tokens that the model consumes directly. It reduces the number of prefill tokens to a minimum, at the price of a trained, model-specific component.

The key point connects with the maths above: compressing what you retrieved by a factor $k$ reduces prefill tokens $\times k$, and therefore TTFT $\sim\times k$ and the attention cost $\sim\times k^2$. It is the lever with the best return when long context is the bottleneck.

3. Prefix caching of stable context

Not all the context changes between requests. System instructions, definitions, recurring reference documents, schemas: these are stable prefixes. Prefix caching stores the already-computed KV cache of those prefixes and reuses it, so that prefill only processes the new part (the query and the specific chunks). If 60 % of your context is stable, you save 60 % of that segment’s prefill on every hit. For it to work, the stable context must go at the start of the prompt (the KV cache is prefix-dependent) and it pays to maximise the hit rate; the hit-rate engineering detail is in Prefix cache hit rate. It combines especially well with RAG: retrieved documents that repeat across sessions get cached once.

4. Semantic caching of answers

A layer in front of the model: if a query is semantically equivalent to one answered before (embedding similarity above a threshold), return the cached answer and skip the model entirely, retrieval, prefill and decode included. In real workloads with long tails of repeated or near-repeated questions (FAQ, support), the saving is enormous because it eliminates the full cost, not just the prefill one. The trap is the threshold: too lax and you serve wrong answers to similar-but-different questions. The design is in Semantic caching for RAG.

5. Structured output and function calling: lean on tools, not on memory

The last lever changes what the SLM depends on. Instead of asking it to know a fact (its weak point), make it call a tool that knows it: a database query, an API, a calculator, a validator. Structured output (forcing JSON conforming to a schema) and function calling turn the SLM into an orchestrator that extracts arguments from the context and delegates the computation or the query. A reasonably capable SLM emits a well-formed tool call far more reliably than it recalls a specific fact. This reduces the pressure on parametric knowledge and on retrieval: for structured, fresh data (prices, inventory, states), querying beats retrieving text and beats memorising. The fundamentals are in Structured output and Function calling.

The complete pipeline

The five levers are not alternatives: they chain together. The flow, with the token counter falling at each step:

The order matters. Retrieving aggressively (high recall) before filtering guarantees that the right material is among the candidates; reranking and compressing afterwards guarantees that only the dense and relevant material pays the prefill toll; caching wraps everything so as not to repeat work. The SLM only sees the final crib sheet, short and ordered.

Implications for on-premise inference

The mental trap to avoid: treating the SLM as a large model with less quality. It is not. It is a different cost profile that rewards a different design. Three practical consequences:

The token budget is a first-class resource. With a large model with a 128K window, “putting in a bit more” is cheap relative to the model. With an SLM, every context token shows up in TTFT and in quality. Treat context size as a quantity to minimise subject to covering the answer, not to maximise.
The investment is worth it precisely because the model is cheap. Reranker, compressor and caches add complexity, but the model they serve is economical enough to run many replicas. The bottleneck shifts from the model to the data pipeline, which is exactly where you want it.
Retrieving does not replace adapting; they combine. For deep, recurring domain knowledge, adapting the SLM with LoRA (see the sibling QLoRA and aggressive multi-LoRA) can put part of the knowledge “into the weights” cheaply, reducing what has to be retrieved. Aggressive RAG and aggressive adaptation do not compete: the first gives freshness and citability, the second gives fluency and domain formatting. A good design uses both.

On the RTX 4090 (24 GB, Ada Lovelace)

The canonical scenario: a quantised SLM (4B–8B in INT4/FP8) fits with room to spare, leaving VRAM for a generous KV cache, essential for prefix caching, and for the reranker (a cross-encoder of a few hundred MB). The LLMLingua-style extractive compressor runs on a separate small model or on CPU. The TTFT calculation above (0.80 s → 0.20 s compressing 4× at ~5,000 tok/s) is representative of this card. The rule of thumb: if TTFT goes above your SLA, the first adjustment is to compress the context, not to change the model.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

With 320 GB and native FP8 prefill is much faster, so the temptation is to relax the token discipline. That is not quite advisable: the lever changes from TTFT to aggregate throughput. Compressing the context not only speeds up each request but frees prefill compute to serve more requests per GPU; compute-bound prefill is exactly the resource that saturates first under load. Here prefix caching and semantic caching, shared across replicas, are what pay off most: at high QPS, the prefill work you avoid by caching is pure throughput you gain. The SLM is still the cheap reasoning engine; the difference is that you now run many in parallel and the data pipeline is what decides how many requests fit.

What we have not covered

Evaluating compression: how to measure that compressing $k=4$ does not throw away correct answers (faithfulness, answer recall over a set of questions with ground truth).
Query-aware versus query-agnostic compression: compressing before or after knowing the question changes what can be cached and what can be discarded.
Chunking and granularity: chunk size interacts with reranking and compression; that is left for the corpus curation post.
Multi-hop and agents: when a question requires several retrieval rounds, the token budget is split across hops and compression discipline becomes critical.

References

Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
Liu, N.F., et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. https://arxiv.org/abs/2307.03172
Jiang, H., et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. https://arxiv.org/abs/2310.05736
Jiang, H., et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024. https://arxiv.org/abs/2310.06839

Aggressive quantisation (state of the art): from 4-bit to ternary

Tue, 09 Jun 2026 02:10:00 +0000

This post is the direct continuation of Quantization for LLM inference, which covers the “solved” regime (FP8, INT4 with GPTQ/AWQ). Read it first: here I assume the scale+zero-point maths, what GPTQ and AWQ do, and the PTQ/QAT distinction. What we add is the sub-4-bit frontier, where scalar post-hoc quantisation stops working and you have to change tools.

TL;DR

There is a sharp dividing line around 4 bits. Above it, quantising is a solved problem: INT8 is indistinguishable from BF16, and INT4 with a good method (AWQ, GPTQ) loses 1-2 MMLU points and little else. The method is still the usual one, take each weight, scale it, round it to a short integer, and it works. Below 4 bits, that method collapses: at 2 bits, naive scalar quantisation can double perplexity. The reason is geometric, each weight has only 4 possible values and the rounding error stops being negligible, and the way out is not “round better”, it is to change the representation. State-of-the-art 2-bit methods (AQLM, QuIP#, QTIP) stop quantising individual weights and quantise vectors of weights against dictionaries (codebooks), and they “whiten” the weight matrix to spread its energy and flatten outliers (incoherence processing). Ternary is something else again: BitNet b1.58, with weights in {-1, 0, +1} (~1.58 bits), is not PTQ, it is a model trained natively under that constraint, and it changes the arithmetic of the matmul from multiplications to additions and subtractions, hitting the compute ceiling and the memory ceiling at once. The mental rule: at ≥4-bit you compress the photo; below 4-bit you have to repaint it.

The analogy: the JPEG that cannot be compressed any further

In the quantization post we used the JPEG with an edge detector to explain INT4. The analogy still holds here, but it has to be pushed to its limit.

A JPEG at quality factor 90 is indistinguishable from the original. At quality 60 you notice it a little, but it is still “the same photo”. At quality 30 the 8×8 blocks appear, the haloes around edges, the banding in gradients. At quality 10 the image is destroyed: you can tell there was a face, but the details have disappeared under the artefacts. And here is the key: no JPEG encoder exists that can compress to quality 10 without those artefacts, because the JPEG algorithm (blockwise DCT + coefficient quantisation) has an information floor below which its own mechanism introduces the noise.

What do you do if you need the photo at that file size and you need it to still look good? You do not compress the original harder. You repaint the photo knowing in advance that it is going to live compressed: an illustrator redraws it with clean lines, a reduced palette, no subtle gradients, an image designed to survive brutal compression. The result at “10 KB” looks infinitely better than the original JPEG crushed to 10 KB, because it is not the same process: one destroys existing information, the other generates new information already adapted to the constraint.

That is exactly the frontier of this post:

Scalar PTQ (≥4-bit) = compressing the JPEG. Up to a certain ratio, it is still the same photo.
State-of-the-art vector PTQ (2-bit: AQLM, QuIP#, QTIP) = a far more sophisticated image codec (dictionaries, decorrelating transforms) that stretches the compressible ratio a bit further before the collapse.
Native ternary (BitNet b1.58) = repainting the photo. You are not compressing an existing BF16 model; you are training a new one that is born ternary.

The map of the frontier, bit by bit

Quantising a model is deciding how many distinct values each weight can take. With b bits per weight there are 2^b possible values. The central question is: past what b is the number of values so small that rounding destroys the model?

Bits	Values/weight	State of the art	Method required	Typical loss vs BF16
8	256	Solved	RTN, SmoothQuant, FP8	~0 (indistinguishable)
4	16	Solved	AWQ, GPTQ	1-2 pp MMLU, +0.1-0.3 PPL
3	8	Small degradation	tuned GPTQ/AWQ, GGUF Q3_K	3-5 pp MMLU
2	4	Serious unless SOTA	AQLM, QuIP#, QTIP (non-scalar)	scalar: collapse; SOTA: 4-8 pp
1.58	3 (ternary)	Native only	BitNet b1.58 (QAT/native training)	n/a (not PTQ)
1	2 (binary)	Research	native, dubious claims	large / no clear methodology

The three transitions that matter:

8 → 4 bits: nothing breaks. With 16 levels per weight and one scale per block of 128, the rounding error is small relative to the dynamics of the weights. GPTQ compensates for the error by propagating it to the neighbouring weights; AWQ protects the ~1 % of salient channels. The model barely notices. This is in the previous post.

4 → 2 bits: the elbow. Two things happen at once here. First, with only 4 levels, the scalar quantiser can no longer represent the weight distribution, which is roughly Gaussian with long tails, without a proportionally enormous rounding error. Second, and more subtly: the quantisation error stops being “small noise that the model absorbs” and becomes structured, systematically biasing the activations. Naive scalar PTQ at 2 bits on a Llama 8B typically doubles perplexity or worse. That is the elbow of the curve.

2 → 1.58 bits: a change of nature. You do not cross this with a better compression method. You cross it by training the model with the constraint from the start. It is a discontinuity: to the left you are doing PTQ, to the right you are doing training.

Why scalar PTQ collapses below 4 bits

The scalar quantiser has a fundamental limitation: it quantises each weight separately, ignoring the fact that the weights of a row or column are correlated and that the error of one could be compensated by another. At 4 bits this matters little; at 2 bits it is lethal. There are three possible attacks, and the state-of-the-art methods use all three.

1. Vector quantisation: dictionaries instead of scales

Instead of mapping each weight to one of 4 values, group the weights into vectors (say, of 8 weights) and map each vector to the nearest entry of a learned dictionary (codebook). If the dictionary has 256 entries, encoding a vector of 8 weights costs 8 bits (the index) → 1 bit/weight, but every “reconstructed value” is a point in an 8-dimensional space chosen to minimise the error over the real distribution of the weights.

The advantage is information-theoretic: a vector dictionary can place its reconstruction points where the weights actually are (in clusters), whereas the scalar quantiser is forced to put its 4 levels on a regular grid, wasting resolution on empty regions. It is the difference between a road map with a uniform grid and one that puts more detail where the cities are.

AQLM (Additive Quantization of Language Models, arXiv:2401.06118) takes this to the extreme with additive quantisation: each weight vector is reconstructed as the sum of several codes from several dictionaries (multi-codebook). It is more expressive than a single dictionary because the number of combinations is the product of the sizes, not the sum. AQLM was one of the first methods to make 2-bit “usable” (not collapsed) on large models, at the cost of an expensive calibration process and specialised inference kernels.

2. Incoherence processing: whitening the matrix

The second attack targets the outliers. The weight matrices of a transformer have a few entries (and a few channels) with a magnitude much larger than the rest. Those outliers dominate the quantiser’s range: if you have to represent a weight of magnitude 8 while the rest are of magnitude 0.5, your scale stretches to cover the 8 and you waste almost all the resolution.

Incoherence processing (the core idea of QuIP and QuIP#) attacks this by multiplying the weight matrix W by random orthogonal matrices on the left and on the right: W' = U W V^T. Because U and V are orthogonal, the operation is invertible and the maths of the product can be undone at inference time by absorbing it into the neighbouring layers (just as AWQ absorbs its scales). But the rotation spreads the energy: an “incoherent” matrix has its values distributed almost uniformly, with no concentrated outliers, because mixing coordinates with a random rotation flattens the distribution (it is, in essence, the central limit theorem acting on linear combinations). A matrix without outliers quantises far better at 2 bits. It is the equivalent of “whitening” a signal before digitising it.

QuIP# (arXiv:2402.04396) combines incoherence processing with E8 lattice codes: rather than an arbitrary dictionary, it uses the E8 lattice (an optimal sphere packing in 8 dimensions, the best known). Quantising vectors of 8 weights against the E8 lattice gives the smallest reconstruction error possible for a given bit density, because E8 is literally the most efficient way to place points in 8D. It is classical coding theory applied to LLM weights.

3. Coding with memory: trellis

QTIP (arXiv:2406.11235) adds the third attack: trellis-coded quantisation. Instead of quantising each vector independently, it models the sequence of weights as a path through a trellis (the same structure as the convolutional codes of telecommunications) and picks the optimal code sequence with the Viterbi algorithm. The intuition: introducing memory between successive quantisations allows correlated errors that cancel out, instead of independent errors that accumulate. QTIP, on top of incoherence processing, improves on QuIP# in quality at 2-3 bits while keeping fast inference kernels.

The idea common to all three: stop quantising scalars and start quantising vectors with dictionaries, and decorrelate the matrix before doing so. None of them is “rounding better”; all three change the representation at the root. That is why, below 4 bits, a flag in vLLM is no longer enough: you need co-design of the quantisation method and the inference kernel.

Native ternary: BitNet b1.58

Here we change continents. Everything above is PTQ: it starts from a trained BF16 model and compresses it. BitNet’s ternary compresses nothing.

BitNet b1.58 (arXiv:2402.17764) trains a transformer from scratch where every weight is constrained to {-1, 0, +1} throughout training. Three values ⇒ log₂(3) ≈ 1.58 bits/weight. Quantisation is not a later step: the linear layers (BitLinear) quantise their weights to ternary in the forward pass of every training step, and gradients flow through a straight-through estimator. The model learns to work with ternary weights. This is QAT taken to the extreme: not a short robustness fine-tune, but the constraint present from the first token of training.

That difference is what dodges the elbow of the curve. PTQ at 2 bits tries to find the best ternary or quaternary approximation of a model that was trained expecting full precision, and that model has “fragile” weights that depend on nuances 2 bits cannot capture. BitNet, by contrast, never had those nuances: its weights were born ternary, so the network distributed its representational capacity in a way compatible with the constraint. It is repainting the photo instead of compressing it.

What changes is not just the memory, it is the arithmetic

The most underrated point about BitNet: with weights in {-1, 0, +1}, multiplication disappears from the matmul. Multiplying an activation x by a ternary weight w is trivial: if w = +1 you add x, if w = -1 you subtract x, if w = 0 you do nothing. The dominant operation of a transformer, the matrix-vector product, goes from a sea of floating point multiply-accumulate (MAC) to integer additions and subtractions.

This matters because it connects to the roofline. As explained in The inverted roofline of small models, LLM inference has two ceilings: memory (HBM bandwidth for loading weights) and compute (tensor core FLOPs). Normal quantisation (INT4, FP8) attacks only the memory ceiling: the weight takes up less space, but to multiply it you dequantise to FP16 and do the same multiplication as always. Ternary attacks both ceilings at once: the weight occupies 1.58 bits (memory) and the operation is an addition instead of a multiplication (compute). That is why BitNet needs its own kernels, bitnet.cpp, that execute the ternary matmul without ever going through FP16; a kernel that dequantised to FP16 in order to multiply would throw away half the advantage.

The honest trade-off: BitNet b1.58 is training from scratch. You cannot “convert your Llama 8B to BitNet”. If you want ternary, you train (or use) a natively ternary model, with everything that implies in pre-training cost and weight availability. Today it is a research line with published models at modest scales, not a drop-in replacement for your current serving.

QAT as the bridge between PTQ and native

Between “compressing post-hoc” (PTQ) and “training natively ternary” (BitNet) there is a middle point: QAT (Quantization-Aware Training). You take an already trained model and do a short fine-tune with the quantisation operations inside the loop, so that it learns to be robust to low bit counts without paying for a full pre-training.

Gemma 3 publishes official QAT variants for exactly this: models that, after the QAT fine-tune, hold up at INT4 with far less quality loss than pure PTQ on the same model. The cost is a training cost (GPU hours to days on an existing model), not an inference cost. For INT4 with QAT you recover almost all the quality; for 2-bit, QAT helps but the terrain is still difficult; for ternary, QAT stops being a “short fine-tune” and becomes native training (BitNet).

The decision hierarchy:

PTQ = default at ≥4 bits. Minutes to hours, without touching training weights. It covers 90 % of production.
QAT = when PTQ loses too much and the difference matters. Low bits (2-3), or sensitive models. You pay for a fine-tune.
Native (ternary) = when you want to go below 2 bits and change the arithmetic. You pay for pre-training. It only makes sense if you control the model from its creation.

The maths that matter: footprint and how many fit

The weight footprint is direct: bytes = (bits/param / 8) × N, with N the number of parameters. For an 8B model:

Level	bits/param	8B footprint	Ratio vs BF16
BF16	16	16.0 GB	1.0×
INT8	8	8.0 GB	2.0×
INT4	4	4.0 GB	4.0×
3-bit	3	3.0 GB	5.3×
2-bit	2	2.0 GB	8.0×
1.58-bit (ternary)	~1.58	~1.6 GB	~10×

(Real ternary takes up slightly more than 1.58 bits/param because you have to pack 5 ternary values into 8 bits, 5 × log₂(3) ≈ 7.92 bits, and because norms and embeddings usually stay at higher precision. The ~1.6 GB figure for 8B is the right order of magnitude.)

How many 8B models fit on an RTX 4090?

An RTX 4090 (24 GB, Ada Lovelace) has 24 GB. Reserving ~4 GB for KV cache and activations leaves 20 GB for weights. How many distinct 8B models fit loaded at the same time:

Level	8B footprint	Models in 20 GB	Comment
BF16	16.0 GB	1	one, with scant headroom left
INT8	8.0 GB	2	two different models
INT4	4.0 GB	5	the solved regime; quality ~lossless with AWQ
3-bit	3.0 GB	6	small degradation already visible
2-bit	2.0 GB	10	only viable with AQLM/QuIP#/QTIP
1.58-bit	~1.6 GB	~12	natively ternary models only

The arithmetic is seductive, from 1 to 12 models on the same card, but it should be read sceptically. Jumping from INT4 (5 models, almost lossless) to 2-bit (10 models) doubles the capacity, but only if you use a state-of-the-art method and accept 4-8 MMLU points. And the jump from 2-bit to ternary (10 → 12) is marginal in memory terms: ternary is not justified by footprint against a state-of-the-art 2-bit, it is justified by the arithmetic (the compute ceiling) and because being native it avoids the quality elbow. If your only metric is “how many GB it takes”, state-of-the-art 2-bit already gives you nearly everything. Ternary is for when you also want the compute saving.

The conceptual curve: perplexity vs bits

Three readings of the curve. One: to the right of 4 bits, the three branches are almost stuck together and almost flat, the solved regime. Two: the red branch (naive scalar PTQ) has a brutal elbow between 3 and 2 bits; that is where it doubles perplexity. The blue branch (state-of-the-art vector PTQ) flattens that elbow, it does not remove it, but it makes it tolerable down to 2 bits. Three: the green point of native ternary is on neither of the two PTQ curves, because it is not obtained by compressing: it is obtained by training, and that is why it can fall below the elbow without paying the quality price that any PTQ pays at that bit density. It is the difference between the crushed JPEG and the repainted photo.

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Three warnings for reading the aggressive-quantisation literature:

“1-bit lossless” almost always comes with small print. Pure binary {-1, +1} (1 bit) loses the ability to represent zero, which matters in transformers (many weights are effectively null). That is why the true state of the art at low density is ternary (1.58 bits), not binary: the zero is worth its extra 0.58 of a bit. When a paper announces “1-bit”, it pays to check whether (a) it really is 1 bit or 1.58 rounded down in the headline, (b) “lossless” is measured in WikiText perplexity (easy) or on reasoning benchmarks (where the collapse shows up), and (c) it compares against a baseline of the same effective size or against a much larger model in order to inflate the advantage.

Flat perplexity ≠ preserved quality. Perplexity on a generic corpus is the metric most forgiving of aggressive quantisation. A 2-bit model can have a perplexity almost identical to BF16 and at the same time drop 10 points on GSM8K or on a code benchmark, because multi-step reasoning amplifies errors that average perplexity does not see. Distrust any sub-4-bit claim that only reports perplexity. As we already said in the quantization post, the loss has to be measured on the target task.

Hardware comparability. The “X times faster” numbers for ternary only apply with the specialised kernels (bitnet.cpp) and on hardware where addition/subtraction arithmetic genuinely wins. On a GPU with tensor cores designed for FP16/FP8, a naive ternary kernel can be slower than a well-optimised INT4, because it wastes the silicon. The ternary advantage is real, but it is a co-design advantage (model + kernel + sometimes hardware), not a flag you switch on over your current stack. Any benchmark that does not specify the exact kernel and hardware is noise.

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace): the practical regime today is still INT4 AWQ for 7-14B models, solved, almost lossless, natively supported. State-of-the-art 2-bit (AQLM/QuIP#/QTIP) is viable and lets you load larger models or more models at once, but it demands the specific kernels of each method and an expensive calibration, and it pays in quality. It makes sense when VRAM is the bottleneck and you accept the trade-off; not as a default. Ternary on a 4090 is experimental: without tensor cores designed for ternary addition and subtraction, the compute advantage is diluted, although the memory saving remains.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8): here the default is FP8 (almost indistinguishable quality, native throughput) or INT4 AWQ for models that do not fit in FP8. State-of-the-art sub-4-bit is for serving enormous models (200B+) when neither FP8 nor INT4 fits with the KV cache headroom you want, at the cost of quality and kernel complexity. Native ternary, today, is a research object rather than a production one: its promise, hitting both roofline ceilings, is greater on CPU/edge (where there are no FP8 tensor cores to exploit) than on an H100 cluster, which already has dedicated FP8 hardware.

The rule of thumb, June 2026: ≥4-bit is settled engineering; state-of-the-art 2-bit is a real lever but with a cost in method and in quality; ternary is an architectural bet, not a deployment tweak.

References

Ma, S. et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58). https://arxiv.org/abs/2402.17764
Egiazarian, V., Panferov, A., Kuznedelev, D. et al. Extreme Compression of Large Language Models via Additive Quantization (AQLM). https://arxiv.org/abs/2401.06118
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., De Sa, C. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. https://arxiv.org/abs/2402.04396
Tseng, A., Sun, Q., Hou, D., De Sa, C. QTIP: Quantization with Trellises and Incoherence Processing. https://arxiv.org/abs/2406.11235
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
Google DeepMind. Gemma 3 QAT (Quantization-Aware Training) models — official blog: https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/
Microsoft. bitnet.cpp — 1-bit ternary inference kernels: https://github.com/microsoft/BitNet

Test-time quantization: quantising on the fly with no calibration dataset

Tue, 09 Jun 2026 02:00:00 +0000

This post is the natural continuation of Quantization for LLM inference, which is worth reading first: that is where GPTQ, AWQ, scale + zero-point and why activation outliers are the central problem all live. Here we are not discussing how many bits to use, but when and with what information the scales are computed: offline against a corpus (PTQ) or on the fly against real traffic (TTQ).

TL;DR

Activation-aware quantisation (AWQ, SmoothQuant) decides which channels to protect by measuring the magnitude of the activations over a calibration dataset in an offline pass, before deployment. The implicit assumption is that this corpus represents future traffic. But activation outliers, the channels with 10-100× the median magnitude that dominate the quantisation error, depend on the input: they change with the domain, the language and the client’s distribution. When real traffic drifts away from the calibration, the fixed scales stop being optimal and quality drops. Test-time quantization (TTQ) removes the corpus and the offline pass: it derives the activation-aware scales at inference time, from the activations that are actually observed, per token or per batch. The trade-off is honest and not minor: it introduces runtime overhead, computing statistics, detecting outliers, recomputing scales at every step, which competes directly with the saving from quantising. On small models that overhead weighs proportionally more, because the forward is short and the fixed per-step costs dominate (the framing is in the inverted roofline for SLMs). TTQ is orthogonal to the format: it is not a competitor to INT4 or FP8, it is a different way of deriving s. It pays off when there is no calibration pipeline, when the traffic distribution is shifting or unknown, and in multitenant settings where no representative corpus exists.

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

A clothes shop has two ways of dressing a customer.

The first is to sell off-the-peg sizes. At some point the factory measured an “average customer”, a mean mannequin built from a population sample, and cut the garments to those measurements. When a customer walks in, you hand them the size that comes closest. It is extremely fast: the garment is already sewn, it just gets handed over. The problem appears when the customer does not look like the average mannequin: if their shoulders are much broader than average, their particular outlier, the standard size pulls or leaves spare fabric, because it was cut to protect other areas. This is offline calibrated PTQ: AWQ measured the importance of each channel over a corpus and fixed the scales once and for all; fast at inference time, but blind to the specific customer.

The second is the tailor who takes measurements on the spot. When the customer walks in, the tailor gets out the tape, measures that customer, spots where their particular bulk is and adjusts the cut to their real anatomy. The result fits better, especially for customers who fall outside the mould. But every customer costs time: measuring, marking, deciding. This is TTQ: the scales are derived on the fly from the activations that the input actually generates.

The analogy holds up in three details:

The average mannequin = the calibration dataset. If the population walking into the shop resembles the mannequin, the sizes work; if not, they fail at the extremes.
Measuring every customer = computing activation statistics per token/batch. A better fit, but a fixed cost paid on every garment.
The broad shoulders = the outlier activation channels. They are precisely the areas where the fit matters and where the generic size gets it most wrong.

The tailor wins when the customers are varied or unknown. The tailor loses when you have a homogeneous population and a mannequin that represents it well: there, paying for measurement on every customer is a waste of time.

The problem TTQ solves: fixed calibration ages with the traffic

Let us recall from the quantization post what AWQ and SmoothQuant do exactly. They do not quantise all channels equally: they identify the ~1 % of channels whose activations have a large magnitude, the salient channels, and protect them by scaling them before quantising. To measure that importance they need to see activations, and they see them over a calibration dataset (128-512 samples, typically WikiText or a slice of the domain) in an offline pass prior to deployment.

The assumption is strong: that the activation distribution of the calibration corpus represents that of production traffic. Two reasons why that assumption breaks:

Activation outliers depend on the input. They are not a fixed property of the model the way weights are. The channel that is an outlier when processing C++ code may not be one when processing conversational Arabic or log JSON. The magnitude and position of the peaks change with the domain, the language and the input format.
Real traffic is rarely the corpus. You calibrate with English WikiText and the client sends you support tickets in Spanish with tables pasted in. The calibration protected the channels that WikiText activated, not the ones that real traffic activates. The scales are suboptimal exactly where the client lives.

The result is distribution-dependent degradation: the quantised model keeps its quality while the input resembles the calibration and loses it as the input drifts away. The most awkward case is multitenant: if you serve clients with different domains from the same quantised model, no single representative corpus exists; any fixed calibration favours some tenants and penalises others.

The TTQ mechanism: measure the real activations and scale on the fly

TTQ (arXiv:2603.19296, March 2026) proposes deriving activation-aware quantisation at inference time, with no offline pass and no calibration dataset. The idea, in its bare conceptual form:

Step 1 — Observe. When the activation tensor X arrives at a linear layer (per token or per batch), cheap statistics are computed over the channels: a measure of central tendency (median or mean magnitude) and one of dispersion per channel. This is the equivalent of AWQ looking at its corpus, but done over the activations that really are coming in right now.

Step 2 — Detect outliers on the fly. With those statistics, the channels whose magnitude shoots up relative to the tensor median are identified; the typical criterion is a threshold of the form “magnitude > k × median”. These are the channels that, if quantised with the same scale as the rest, blow up the error.

Step 3 — Derive scales and segregate. For normal channels a scale is computed that exploits the range; for the outliers a different treatment is applied, a scale of their own, or keeping them at higher precision, in the style of on-the-fly mixed precision. It is the same philosophy as LLM.int8() (segregating outliers to FP16) or AWQ (scaling salient channels), but with the threshold and the scales recomputed over the current input, not frozen since calibration.

Step 4 — Quantise and multiply. With the fresh scales, the tensor is quantised and the GEMM runs. The activations entering the next layer compensate for the rescaling, just as in AWQ, so that the maths cancels out.

The key difference with AWQ is not in what is done (protecting activation outliers) but in when and against what: AWQ decides once, offline, against a corpus; TTQ decides at every step, on the fly, against real traffic. It is the transfer to inference of the “test-time” idea: adapting the computation to the specific sample in front of you instead of to a precomputed average.

The maths that matter

The error of quantising an outlier with the wrong scale

Let us recall affine uniform quantisation from the base post: an integer code q = round(x/s) - z with scale s and zero-point z, and reconstruction x̂ = s·(q + z). For a b-bit quantiser with a symmetric range, the scale that covers a tensor of maximum magnitude M is approximately s = M / (2^{b-1} - 1). The rounding error of each element is bounded by half a scale: |x - x̂| ≤ s/2.

Here is the outlier problem. The scale s is chosen to cover the largest value in the group. If one channel has 30× the median magnitude and you share a single scale with the rest of the tensor, that magnitude rules: M is the outlier, so s is inflated 30× relative to what the majority would need. The absolute rounding error of the normal values rises proportionally.

A concrete calculation. Take a group where the median magnitude is 1.0 and an outlier channel is 30.0, quantised to INT4 (b = 4, levels ±7):

With a shared scale, s = 30 / 7 ≈ 4.29. The rounding error of a typical value (magnitude ~1) is up to s/2 ≈ 2.14. In other words, the error on the normal values is of the order of their own value: the outlier has destroyed the resolution of everything else. Relative error of a value of magnitude 1: up to ~214 %.
Segregating the outlier (pulling it out to FP16 or giving it its own scale) and quantising the rest with M = 1, s = 1/7 ≈ 0.143. The error of a typical value drops to s/2 ≈ 0.071, ~7 % relative. Thirty times less error on the majority of the group’s weights.

That is the entire reason for the existence of activation-aware quantisation: to detect and treat separately the ~1 % of channels that, if not segregated, hijack the scale. AWQ does it against the corpus; TTQ does it against the real input. And if the channel that is an outlier in production was not an outlier during calibration, AWQ did not protect it: it quantised real traffic with the inflated scale from the case above. That is where TTQ gains precision.

The overhead: the cost of measuring at every step

The price is symmetric. Computing the per-token statistics, per-channel magnitudes, median or percentile, outlier threshold, scales, means reductions over the activation tensor that did not exist in the forward pass with frozen scales. Let us call:

T = forward time per token with fixed scales (static PTQ), in µs.
Δ = extra per-token cost of deriving the statistics and scales on the fly, in µs.

The relative overhead is simply:

$$\text{overhead} = \frac{\Delta}{T}$$

The key point is that Δ is relatively fixed per step (it depends on the number of channels and layers, not on how much “useful” work the model does), while T scales with the size of the model. That is why the ratio behaves very differently depending on the model:

Large model (say, 70B): T is large, every forward moves tens of GB of weights from HBM. If Δ ≈ 8 µs and T ≈ 800 µs, the overhead is 8/800 = 1 %. Negligible against the saving from quantising.
SLM (say, 1B): T is small, the forward per token is short. With the same Δ ≈ 8 µs and T ≈ 60 µs, the overhead is 8/60 ≈ 13 %. No longer negligible: it eats a good part of what you gained by quantising.

This connects directly to the inverted roofline for small models: on an SLM the fixed per-step costs (kernel launches, synchronisations, overheads that do not scale with the model) weigh proportionally more, because there is less useful work to spread them over. TTQ’s Δ is exactly one of those fixed costs. Going per-batch instead of per-token amortises Δ across all the tokens of the batch and lowers the relative overhead, at the cost of coarser scales; it is the first parameter to touch.

The uncomfortable conclusion: TTQ gives away robustness to distribution but spends part of the acceleration budget on measuring, and in the regime where acceleration is scarcest, the SLMs, the ones most often deployed at the edge, is where that spending hurts most. It is not free; it is a change of currency.

A note of methodological scepticism: arXiv:2603.19296 is from March 2026, very recent, and as of the date of this post there are no broad independent reproductions. Whatever speedup and quality figures circulate should be taken with the same caution as any number without published methodology: what hardware, what batch size, what real measured Δ, against what baseline (well-calibrated or badly calibrated PTQ), in what domain? The conceptual argument, robustness to distribution in exchange for per-step overhead, is solid; the concrete multipliers are pending validation.

What TTQ is NOT: marking it off from the rest of the zoo

TTQ is easily confused with neighbouring techniques. The distinction that matters is that TTQ is the how of deriving the scales, not the format nor the moment in training.

Technique	When the scales are fixed	Needs a calibration corpus	Touches training	Is it a format
Static PTQ (GPTQ, AWQ)	Offline, before deployment	Yes	No	No (it uses INT4/INT8)
QAT	During training	No (training data)	Yes (retrains)	No
FP8 end to end	At runtime, but simple per-tensor scales	Minimal / none	No	Yes (E4M3/E5M2)
TTQ	At runtime, activation-aware per token/batch	No	No	No (orthogonal to the format)

The four distinctions, one by one:

Against static PTQ (GPTQ/AWQ). Same goal (protecting outliers), the same possible format (INT4), but PTQ freezes the decisions offline against a corpus and TTQ recomputes them on the fly. TTQ is, in a sense, “AWQ without the calibration phase, paid for at runtime”.
Against QAT. QAT puts quantisation inside the training loop so that the model learns to be robust to it; it costs a retrain. TTQ does not touch training: it operates on an already trained model, at inference. They are attacks at opposite moments of the pipeline.
Against FP8 end to end. FP8 is a format with its own logarithmic range; its “dynamic scaling” computes a simple per-tensor scalar at runtime, but it does not do activation-aware per-channel outlier detection. TTQ could, conceptually, derive scales on the fly for an FP8 or INT4 quantiser: it is orthogonal to the format.
TTQ is orthogonal to the format. It decides how to obtain s, not how many bits you store q in. You can imagine “TTQ over INT4” or “TTQ over FP8”. What defines TTQ is the source of the scale, real activations on the fly, not the width of the code.

When it pays off (and when it does not)

TTQ is not a universal replacement for AWQ. It is a tool for a specific deployment profile. It pays off when:

You have no calibration pipeline. You want to deploy a quantised model now, without assembling the calibration dataset, running the offline pass or validating that the corpus represents the traffic. TTQ cuts out that whole phase: you load the model and serve.
The traffic distribution is shifting or unknown. An assistant that one day receives code and the next legal contracts in another language. No fixed calibration covers both well; on-the-fly adaptation follows the distribution without recalibrating.
Multitenant with no representative corpus. You serve the same model to clients with disparate domains. No single corpus represents all of them; any fixed calibration creates winners and losers among tenants. TTQ adjusts to each input, whichever tenant it comes from.

It does not pay off when:

You have a stable domain and a good calibration corpus. If your traffic is homogeneous and representative, offline AWQ gives you the same quality with zero runtime overhead. Paying Δ on every token to relearn what a corpus already captured is waste.
You serve SLMs with a tight latency SLA. This is exactly the case where Δ/T is high. If the model is small and TPOT matters, the overhead of measuring can wipe out the gain from quantising. Measure your real Δ before assuming it adds up.
The batch is large and compute-bound. With high concurrency the forward is no longer memory-bound and the cost of the extra reductions competes worse; at the very least, amortise Δ per batch.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The natural case for the 4090 is the SLM, Qwen 3 1.5B, Llama 3 8B AWQ-INT4, serving at low concurrency. That is precisely the regime where TTQ is riskiest: T per token is small and the 4090 has no accelerated native FP8 (we discussed this in the quantization post), so TTQ’s extra reductions run on CUDA cores competing for the same time. Here the question is not “does it improve quality?” but “does the overhead leave me an acceptable TPOT?”. If the traffic is homogeneous, offline AWQ wins on simplicity and latency. TTQ only justifies its Δ if the input distribution is genuinely unpredictable and the degradation from fixed calibration is measurable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here the calculation partly flips. With large models T is high and Δ/T drops to the low single-digit percentage range, so TTQ’s overhead is more digestible. The strong use case is multitenant: a cluster serving a large model to clients with heterogeneous domains, where no calibration corpus satisfies everyone. There, TTQ’s robustness to distribution has real value and the overhead is diluted in a big forward. Even so, on an H100 with native FP8, the baseline to beat is demanding: static FP8 barely loses quality (see the table in the quantization post) and costs nothing at runtime. TTQ has to show that its robustness gain on the outlier tenants outweighs what it gives away in overhead. With a paper from March 2026 and no reproductions, that demonstration is pending.

What we have not covered

The memory cost of the on-the-fly statistics: per-channel buffers, their impact on the footprint and on cache pressure.
Interaction with continuous batching: how scales are derived when a batch mixes requests from different domains in the same step.
TTQ + speculative decoding: whether the draft and the target derive scales on the fly separately, and how that affects the acceptance rate.
Numerical stability: what happens when a batch contains a one-off extreme outlier that inflates the scale for every token of that step.

References

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly (March 2026). https://arxiv.org/abs/2603.19296
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024). https://arxiv.org/abs/2306.00978
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023). https://arxiv.org/abs/2210.17323
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023). https://arxiv.org/abs/2211.10438
Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022). https://arxiv.org/abs/2208.07339

Device-native architectures: fine-grained MoE and pre-attention router

Tue, 09 Jun 2026 01:50:00 +0000

This post belongs to the series on inference performance in small models. It is the architectural face of a problem we have already looked at from the compute-regime side (the SLM’s inverted roofline) and from the weight-loading side in From disk to HBM. Here the question is different: what if, instead of adapting a large model to the device, we design the model for the device from the first commit?

TL;DR

The default move to get an LLM onto a laptop, a phone or an edge box is to take a dense model built for cloud and compress it: distillation, pruning, quantisation. It is a gesture of reduction: you start from something big and take things away. SmallThinker (arXiv:2507.20984, SJTU IPADS + Zenergize AI) argues for the reverse gesture, designing from scratch, and articulates it in three pieces. First: fine-grained MoE, many small experts with very few activated per token, so that total parameters N (the capacity) decouple from activated parameters A (the compute cost per token). Second: sparse FFN, ReLU-style activation sparsity inside each block, which adds a second level of sparsity on top of the first. Third: a pre-attention router that predicts which experts will be needed before running the attention block and launches the prefetch of those weights from SSD/flash in parallel with attention compute, hiding the storage latency, which is the real bottleneck when the model does not fit entirely in RAM. The authors report SmallThinker-4B-A0.6B and SmallThinker-21B-A3B exceeding ~20 tok/s on consumer CPUs with Q4_0, using ~1 GB and ~8 GB of RAM. The numbers are interesting and the direction is right; the evaluation methodology and the quality cost of activating so little deserve scepticism, and that is what the last part is for.

The analogy: the librarian who anticipates your request

Picture a huge library with a small reading room. You are sitting in the room at a single desk: only a few books fit on it at a time (that is RAM). The bulk of the collection is in the back room, on long shelves that are slow to walk (that is the SSD/flash). And there is a librarian.

The naive method: you read, you reach a point where you need a specific book, you request it, and then the librarian gets up, walks to the back room, looks for it and comes back. Meanwhile, you wait with the page open, making no progress. Every time you need a new book, you pay the full trip to the back room. The reading room spends most of its time waiting, not reading.

The SmallThinker method: the librarian is clever and gets ahead of you. While you are still reading the chapter’s table of contents, working out what it is about and connecting ideas, which in the model is the attention block, they have already glanced over your shoulder, predicted which three or four books you are going to ask for and gone to the back room to fetch them. By the time you finish the contents and formulate the request, the books are already on your desk. You have not waited: the trip to the back room happened in parallel with your reading of the contents.

The analogy holds on four details:

The small desk is RAM; the slow back room is the SSD/flash.
The books are the MoE’s experts: only a few are on the desk at any moment.
Reading the contents is the attention block; requesting and using the books is the FFN/expert block.
The librarian who predicts and gets ahead is the pre-attention router: the prediction is made first, and the trip to fetch (the prefetch) overlaps with reading the contents (attention).

The quantitative question running through the whole post is: does the librarian get back in time? The wait is only hidden if the trip to the back room takes less than your reading of the contents. That is the condition t_{\text{attention}} \ge t_{\text{prefetch}}, and we will do it with numbers.

Compressing a dense model vs. designing for device

It is worth laying out the two approaches plainly, because they are not degrees of the same thing: they are different philosophies.

Approach A — compress a dense model built for cloud. You start from, say, a dense 7B–14B model trained to run on an RTX 4090 (24 GB, Ada Lovelace) or on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). To fit it into a device you apply three levers, each with its own post: distillation (you train a small student that imitates the teacher), pruning (you remove weights or whole structures) and aggressive quantisation (you go down to 4 bits or fewer). The resulting model is still dense: all its parameters are activated on every token. You have reduced the number of parameters, but the compute pattern is the cloud’s, only smaller.

Approach B — design for device from scratch. Here the device’s constraints enter the architecture, not a later compression phase. The constraints are three and very concrete:

Weak compute. A laptop CPU or a mobile SoC does orders of magnitude fewer FLOPs than a datacenter GPU. This pushes you to minimise the activated parameters per token, not the total ones.
Little RAM. Tens of GB do not fit. This pushes you to keep resident only what is essential and to stream the rest.
Slow storage. The SSD or flash you are forced to stream from has far less bandwidth than a GPU’s HBM. This turns storage I/O into the bottleneck, and pushes you to hide it.

SmallThinker is approach B taken to the detail: each of those three constraints has an architectural answer. Weak compute is attacked with fine-grained MoE + sparse FFN (minimise A). Scarce RAM is attacked with streaming from SSD (resident ≈ A + cache, not N). Slow storage is attacked with the pre-attention router (hide the I/O behind attention). It is no accident that the three pieces fit together: each one solves a constraint, and together they reinforce each other.

An important nuance, so as not to fall into the hype: approach B is neither free nor universally superior. It requires training a new model (you do not reuse existing weights), and the quality ceiling of a model with a very small A is intrinsically bounded, as we will see. The argument is not “B always wins”, but “for the device regime, B attacks the right bottlenecks, and A only attacks them in passing”.

Two levels of sparsity

The central capacity idea is old and well understood in MoE: separate capacity from compute cost. In a MoE, the model has N total parameters spread across experts, but for each token only A parameters are activated (those of the top-k experts the router picks). Compute cost per token scales with A; knowledge capacity scales with N. SmallThinker applies this idea at two superimposed levels.

Level 1 — fine-grained MoE. “Fine-grained” means many small experts instead of few large ones, with very few activated per token. Instead of, say, 8 experts of which you activate 2, you have dozens of experts of which you activate a handful. With smaller experts, the same A is spread across more possible combinations, which gives the router fine granularity and keeps A very low relative to N. The result is an aggressive N/A ratio: a lot of capacity, very little compute per token.

Level 2 — sparse FFN (ReLU-style activation sparsity). This level is orthogonal and operates inside each FFN. With a ReLU-style non-linearity, a large fraction of the intermediate layer’s neurons produce exactly zero for a given token. A neuron that comes out at zero contributes nothing to the output: its matrix-vector multiplication can be skipped. This is activation sparsity: predictable token by token, and exploitable so as not to load or multiply the weight rows/columns corresponding to inactive neurons. It is the same phenomenon exploited by work such as Deja Vu or PowerInfer; SmallThinker builds it in from the factory by choosing activations that favour it.

The combined effect, in one sentence: large N (capacity), tiny A (compute cost per token ≈ proportional to A), and on top of that, inside that A, a fraction of the multiplications is saved through activation sparsity. It is sparsity on top of sparsity.

The pre-attention router: predict and prefetch

Here is the paper’s specific piece, and the one that gives the post its name. The problem it solves is one of I/O scheduling, not of quality.

When the model does not fit entirely in RAM, the expert weights live in SSD/flash and are loaded on demand. The naive flow of a MoE layer is sequential: you run attention, then the router decides which experts are needed, then you load those experts from SSD (waiting), then you run the FFN of those experts. The loading step is pure waiting: the CPU is blocked waiting for bytes from the SSD. In the device regime, where the SSD is slow, that waiting time dominates the decode step.

The pre-attention router breaks the sequentiality by inverting the order of the decision. The observation is that the router does not need this same layer’s attention output to make a reasonable prediction of which experts will be needed: it can predict it from the state it already has before running attention. So:

Before running the layer’s attention block, the router predicts the experts that will be needed.
It launches the prefetch of those experts from SSD/flash asynchronously.
In parallel, the CPU runs the attention block, which is pure compute and does not need the SSD.
When attention finishes, the prefetched experts are (ideally) already in RAM, and the FFN proceeds without waiting.

Storage I/O has been overlapped with attention compute. It is exactly the librarian who goes to the back room while you read the contents.

The hiding condition is the inequality above: the prefetch is completely hidden if and only if

$$t_{\text{attention}} \;\ge\; t_{\text{prefetch}}.$$

If attention takes longer than loading the experts, the load is free (it was already done). If the experts are too large or the SSD too slow, t_prefetch > t_att and a wait bubble equal to t_prefetch − t_att appears. That is why the design needs A to be small (small experts → fewer bytes to prefetch → low t_prefetch) and the grain to be fine: the two things level 1 of sparsity does are not only about saving FLOPs, they are about making the prefetch fit underneath attention.

The maths that matter

Memory footprint: resident N vs. A + cache

The parameter that decides whether the model fits is how much you have to keep resident in RAM at once.

Everything in RAM. If you require all the experts to be loaded, the footprint is \approx N (all parameters, multiplied by bytes/parameter according to the quantisation). For a 21B this is prohibitive on a device.
Streaming from SSD. If you only keep the active experts resident plus a cache of the recent/likely ones, the footprint drops to \approx A + \text{cache}. The weights that are not in RAM live on SSD and get prefetched when needed. This is where the real saving is: the resident set scales with A, not with N.

The non-expert part of the model (embeddings, attention, router, layernorms) is always resident, but in a fine-grained MoE the bulk of N is in the experts, so the approximation resident ≈ A + cache + dense_part is a good one.

The prefetch calculation, with numbers

Let us put numbers on the analogy. Suppose a consumer SSD at 5 GB/s of sequential read and a quantised expert of size X MB. The time to load one expert is

$$t_{\text{1 expert}} = \frac{X \text{ MB}}{5000 \text{ MB/s}} = \frac{X}{5000}\ \text{s} = \frac{X}{5}\ \text{ms}.$$

Let us make X concrete. In SmallThinker-4B-A0.6B with Q4_0 (~0.5 byte/param effective counting block overhead), a small expert of, say, 4M parameters weighs \approx 4\text{M} \times 0.5 = 2 MB. Loading it costs t_{\text{1 expert}} = 2/5 = 0.4 ms.

Now the scheduling question: if the layer’s attention block takes Y ms, how many experts can I prefetch while attention runs? The number is

$$n_{\text{prefetch}} = \left\lfloor \frac{Y}{t_{\text{1 expert}}} \right\rfloor = \left\lfloor \frac{Y \cdot 5}{X} \right\rfloor.$$

With Y = 2 ms of attention and X = 2 MB per expert: n_{\text{prefetch}} = \lfloor 2 \times 5 / 2 \rfloor = 5 experts. That is, within that layer’s attention window the SSD manages to bring in 5 experts. If the layer’s top-k activates ≤ 5 experts, the prefetch hides them all and t_prefetch ≤ t_att: zero load latency. If the layer needed 8 experts, you would bring 5 for free and pay the load of the remaining 3 as a bubble: (8-5) \times 0.4 = 1.2 ms of waiting per layer. Hence why the design wants fine grain with a small top-k: to fit underneath the attention window.

Two critical observations about this calculation:

The 5 GB/s is idealised sequential read. The experts are scattered across the disk; random 4K reads on a consumer SSD are far slower. The effective bandwidth can be a fraction of the nominal one, which reduces n_{\text{prefetch}}. Any methodology reporting tok/s ought to say whether it measures with experts pre-sorted on disk or with realistic access.
The attention window Y shrinks with short context and at the start of generation. With short prompts, attention is cheap and may not cover the prefetch; the advantage of the overlap grows with longer sequences. Another detail an honest benchmark should break down.

Weight footprint: why they report ~1 GB for a 4B

Let us do the 4B calculation in Q4_0. Quantisation to 4 bits ≈ 0.5 byte/param, plus a small overhead of per-block scales (Q4_0 adds one FP16 scale every 32 weights, ~0.56 effective byte/param). So:

$$4\text{B} \times 0.5\ \text{B/param} \approx 2\ \text{GB}.$$

That is, the full model in Q4_0 takes up ~2 GB on disk. But the authors report ~1 GB of RAM. A contradiction? No, and understanding why is understanding the design:

Not all experts are resident. Only the activated ones (A = 0.6B) plus a cache fit in RAM; the rest lives on SSD and is streamed. 0.6\text{B} \times 0.5 \approx 0.3 GB of active experts, plus the dense part (attention, embeddings, router) and a cache of hot experts.
Sparse FFN reduces the work and the useful resident set. The neurons that come out at zero do not need to be materialised for that token.

Adding up active experts + dense part + a reasonable cache, ~1 GB is plausible. But mind the nuance: ~1 GB is the resident set in RAM, not the total footprint in storage, which is still ~2 GB on SSD. Conflating the two, reporting a bare “1 GB”, is misleading if the reader understands “the model takes up 1 GB”. It takes up 2 GB; it keeps 1 GB in RAM. The distinction matters for a device with 2 GB of free storage: there it does not fit.

Analogously, SmallThinker-21B-A3B: 21\text{B} \times 0.5 \approx 10.5 GB on disk; 3\text{B} \times 0.5 \approx 1.5 GB of active experts, and the ~8 GB of RAM reported includes active experts + a generous cache + the dense part. The large cache is what takes it from 1.5 to ~8 GB: you keep many hot experts resident so as not to hit the SSD constantly.

The quality cost: the necessary scepticism

All the machinery above reduces the compute per token to \approx A. But A = 0.6B activated is very little. This is where the enthusiasm needs a brake:

Bounded reasoning capacity. A model that activates 0.6B parameters per token has, per token, the compute power of a 0.6B model, not of a 4B one. The total capacity N=4B helps to store more knowledge (more specialised experts), but the processing of each token is still limited by A. For tasks requiring composition and intensive multi-step reasoning, this is a real ceiling, not a detail.
The router is a single point of failure for quality. If the fine-grained router picks the wrong experts, and with fine grain there are more decisions to take, quality drops without any speed metric reflecting it. The pre-attention router makes this worse: it predicts the experts before seeing attention, with less information than a post-attention router. The authors ought to report how much quality is lost by predicting early (mismatch between the prefetched expert and the expert a post-attention router would have chosen).
The ~20 tok/s needs small print. On which CPU exactly? With what context and generation length (the advantage of the overlap depends on Y)? Cold start included or steady state? Did the SSD have the experts pre-sorted sequentially? An “exceeds 20 tok/s” without those conditions is a marketing number, not a methodological one.
A fair comparison. The right question is not “is it fast?”, but “at equal quality on an independent benchmark, is it faster or smaller than an equivalent compressed dense model?”. That requires evals the reader can reproduce, not just tok/s on the authors’ machine.

None of this invalidates the direction. Designing for device is, conceptually, the right approach: it attacks the real bottlenecks (compute, RAM, I/O) in the architecture instead of patching them afterwards. But “20 tok/s in ~1 GB” is a claim about efficiency, and efficiency only means something anchored to a level of quality measured honestly. Until that anchor is clear, the right amount of scepticism is high.

Implications for on-premise and edge inference

The SSD becomes part of the inference hierarchy. In cloud, the hierarchy is HBM → RAM. On device, the SSD/flash enters as one more level, and its bandwidth and random-access latency become first-order performance parameters. This connects with From disk to HBM: cold start and weight streaming stop being just a startup problem and become part of the steady state.
The heterogeneous edge box starts to make sense. In a mixed environments pattern, a device-native model like SmallThinker runs on the NUC/edge with CPU and SSD, serving locally, while the heavy work stays on the central cluster. The pre-attention router is what makes a GPU-less edge box viable.
Capacity planning changes axes. As discussed in Inference capacity planning, on device the resource to plan is not VRAM but the triad resident-RAM / SSD-bandwidth / CPU-FLOPs. A model with small A and overlapped prefetch moves the bottleneck from “does it fit in RAM?” to “does the SSD feed the prefetch in time?”.

Conclusion

SmallThinker is, above all, a change of question. Not “how do I shrink this cloud model so it fits on the device?” but “what would the model look like if I designed it for the device from the first parameter?”. The answer, fine-grained MoE to decouple N from A, sparse FFN to save inside A, and a pre-attention router that hides storage I/O underneath attention, attacks the device’s three constraints (compute, RAM, I/O) in the architecture, not in a later compression phase. The key condition, t_att ≥ t_prefetch, explains why the pieces fit: fine grain does not only save FLOPs, it makes the prefetch fit underneath attention. The reported numbers (~20 tok/s, ~1 GB / ~8 GB of RAM) are promising and the direction is solid; the cost of activating so little and the lack of methodological detail on quality call for caution. Designing for device is the right bet; measuring it honestly is the unfinished business.

References

Equipo SmallThinker (SJTU IPADS + Zenergize AI). SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment. arXiv:2507.20984. https://arxiv.org/abs/2507.20984
Repositorio oficial SmallThinker: https://github.com/SJTU-IPADS/SmallThinker
Self-Speculative Decoding for On-device MoE Acceleration. ACM The Web Conference (WWW) 2026. doi:10.1145/3774904.3792218. https://doi.org/10.1145/3774904.3792218
Liu, Z. et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML 2023. https://arxiv.org/abs/2310.17157
Song, Y. et al. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (sparse activation + hot/cold experts). SJTU IPADS, 2023. https://arxiv.org/abs/2312.12456

Self-speculative decoding: the model that gets ahead of itself

Tue, 09 Jun 2026 01:40:00 +0000

This post is the direct companion to Speculative decoding: the secretary who types ahead. There, draft and target are two different models; here they are the same model at two depths. Read that one first: we take rejection sampling, the 1/(1-α) ceiling and the speedup formula as known, and all we change here is what the draft is.

TL;DR

Classic speculative decoding demands a pair: a cheap draft model proposes γ tokens and an expensive target verifies them in a single parallel forward pass. On large models the draft can be 1 % of the target and fit comfortably. On small models (SLMs, 1B–8B) that recipe breaks on two fronts: a draft that is 1/10 of a 3B is a 0.3B that barely gets anything right (α collapses), and loading a second model, however small, doubles the pieces to maintain and eats VRAM that is not spare on a 4090 or on device. Self-speculative decoding solves both: the draft is the model itself, run shallowly. A model with L layers produces draft tokens by exiting at an intermediate layer k < L (early-exit) or by skipping a subset of layers (layer-skip), and then verifies those tokens with the full forward pass through the L layers. Because draft and verify share weights and share the KV cache of the common layers, the extra memory cost is zero: there is no second model, no second KV cache, nothing new to load. The price is that the early-exit draft is more expensive than a tiny external draft (it walks k/L of the model instead of 1 %), so the relative cost c goes up. The honest trade-off: with a well-trained dedicated draft (EAGLE-3) that fits in memory, its α is usually higher and it wins; self-spec wins when there is no trained draft, when it does not fit, or when you are on device.

The analogy: the chess player who plays by eye and then calculates

A good chess player does two things with the same brain. First they look at the board and, by eye, in half a second, propose a move that “looks right”: pattern intuition, fast recognition, the shallow layers of judgement. Then, before moving, they calculate properly: three moves ahead, the opponent’s replies, the tactical lines. That deep calculation confirms the intuition or corrects it.

The decisive part is that it is the same person acting as drafter and as reviewer. They do not hire a second, weaker player to guess the move so they can validate it; that would be classic speculative with an external draft. Here the fast draft and the slow verification come out of the same brain, traversed at two depths.

The analogy holds point by point:

The glance by eye is the early-exit forward pass: the model walks only the first k layers and emits a draft token. Fast, approximate.
The deep calculation is the full forward pass through the L layers, which verifies the draft with exact rejection sampling.
That it is the same person is the reuse of weights and KV cache: the k shallow layers of the draft are literally the same as the first k layers of the verify; what has already been computed is not recomputed.
That the final move is identical to the one the player would have chosen by always calculating deeply is the rejection sampling guarantee: output quality does not degrade (the proof is in the speculative post).

Why the external draft does not fit small models

Let us revisit the cost of classic speculative with two numbers. The speedup depends on the acceptance rate α (how often the draft is right) and on the relative cost c = T_draft / T_target. A useful draft needs high α and low c at the same time. On large models that is reachable: a 1B draft for a 70B target has c ≈ 0.015 and, if well distilled (EAGLE), α > 0.8. The product pays off.

On a small model the balance breaks:

The proportional draft is useless. If you want c ≈ 0.1 for a 3B target, your draft is around 0.3B. A generic 0.3B has a distribution so different from the 3B that α falls into the 0.3–0.5 range. And 1/(1-α) with α = 0.4 is a ceiling of 1.67 tokens/step: not even infinite γ gets you more. The prize evaporates.
Loading a second model doubles the pieces. Even if the draft is small in VRAM, it is another checkpoint to version, quantise, validate and serve, and it has its own KV cache. On an RTX 4090 (24 GB, Ada Lovelace) with a quantised 8B and a long context, the KV cache is already tight; adding a second model and its cache can force you to lower concurrency or maximum context. On device (a phone, a NUC, an edge box) there is simply no room.
A trained draft does not always exist for your exotic or fine-tuned model. EAGLE needs the draft trained on-policy against that specific target (see knowledge distillation). If your SLM is your own fine-tune, there is no official published draft.

Self-speculative attacks all three at once with one idea: do not bring a second model; use the first one at half depth.

The mechanism: early-exit as draft, full forward as verify

A transformer with L layers transforms the hidden state layer by layer at each position: h_0 → h_1 → ... → h_L, and the LM head projects h_L to logits. The observation that enables everything: h_k for k < L is already a reasonable hidden state. If you push it through the same LM head (or through a dedicated lightweight head), you get an output distribution that is “premature” but often correct for easy tokens. That is the source of the draft.

The self-speculative iteration has the same structure as classic speculative (draft, verify, accept/reject) but both roles are the same model:

Step 1 — Shallow draft. To produce γ draft tokens, the model walks only the first k layers (or a subset of layers in the layer-skip case) and applies the LM head. Each draft token costs ≈ k/L of a full forward pass. We call c = k/L the relative cost of the draft. The γ drafts are generated autoregressively at this reduced cost.

Step 2 — Full verify. The model runs a single forward pass through the L layers over prompt + x_1...x_γ. Because attention is causal it obtains p(·|prompt, x_<i) for each position, exactly as in classic speculative.

Step 3 — Accept/reject. Rejection sampling identical to the previous post: tokens are accepted from left to right, the first divergence is corrected by sampling from the residual norm(max(0, p−q)), and if all γ are accepted the bonus token is appended. Output quality is exactly that of the full model.

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

Here is the key difference against an external draft. When the model drafts by walking layers 0..k, it computes and stores the KV cache of those k layers for the prompt tokens and the drafts. When the full verify comes along, layers 0..k of the L-layer forward pass are bit for bit the same operations on the same weights that the draft already performed. There is no need to recompute them: the verify reuses the KV cache the draft left for layers 0..k directly, and only genuinely computes the missing layers k..L.

That has two consequences:

Zero extra memory. There is no second KV cache. The KV of the common layers is a single one, shared between draft and verify. Contrast that with vanilla SD, where the draft has its own complete cache (see KV cache).
Partially reused compute. The verify only pays for the “new” layers k..L for tokens that already went through the draft. The full forward pass is not as expensive as L suggests, because the first k layers come from the cache.

The families (state of play 2026)

There is no single way of doing self-speculative. They differ in which layers are skipped and in whether training is needed.

Family	Year / venue	How it picks what to skip	Training?	Distinctive idea
LayerSkip (Elhoushi et al.)	2024, arXiv:2404.16710	Early-exit at a fixed layer `k`; a single LM head serves every exit	Yes — layer dropout + early-exit loss in train/fine-tune	A single model trained to do both draft and verify; reuses partial compute
SWIFT	ICLR 2025 (OpenReview EKJhH5D5wA)	Selects which layers to skip on the fly, without touching weights	No — plug-and-play on the given model	Training-free self-spec: optimises the set of skipped layers at runtime
CLaSp	2025, arXiv:2505.24196	Dynamic in-context layer skip: the pattern of skipped layers adapts to the context	No (dynamic at inference)	The skip is not fixed; it changes with what is being generated
ConfLayers	2026, arXiv:2604.14612	Skips layers according to the confidence of the intermediate state (adaptive per token)	No (confidence criterion)	Variable depth: easy tokens exit earlier, hard ones go deeper
Saguaro	2025–26	Asynchronous formulation: the draft keeps speculating in parallel while verification runs	Depends on the variant	Overlaps draft and verify in time instead of alternating them
SSD for on-device MoE	ACM Web Conf. 2026, doi 10.1145/3774904.3792218	Self-spec exploiting MoE sparsity (few active experts per token)	MoE-specific variant	The shallow draft activates even fewer experts; fits MoE on device

Three operational readings of the table:

The axis that matters most is training yes/no. LayerSkip gives the best α because the model learns to be a good shallow draft (with early-exit loss the intermediate layers are explicitly trained to predict well). But it demands a fine-tune. SWIFT, CLaSp and ConfLayers are training-free: worse α, but they apply to any already-trained model without touching anything. For an SLM you do not control, training-free is the realistic option.
Adaptive skip (CLaSp, ConfLayers) raises α because it fits the draft’s depth to the token: it spends little on the easy ones and more on the hard ones, instead of a fixed k. In exchange, the effective c stops being constant.
Saguaro attacks something else: it does not raise α, it overlaps draft and verify time. It is orthogonal to the rest and can be combined.

The maths: same framework, different `c`

We reuse the apparatus of the speculative post without changing a letter. With α the acceptance rate and γ the number of drafts:

$$E[\text{tokens per step}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad \text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

And the algorithmic ceiling is the same: lim_{γ→∞} = 1/(1-α). The only thing that changes in self-speculative is the value of c: it is no longer the size ratio of two models, but c = k/L, the fraction of layers the early-exit draft walks.

Numerical example: self-spec with L=32, exit at k=8

Take an SLM with L = 32 layers that exits at k = 8 for the draft: c = k/L = 8/32 = 0.25. Suppose α = 0.7 (reasonable for early-exit on conversational tokens) and γ = 4.

Expected tokens per step: (1 − 0.7⁵) / (1 − 0.7) = (1 − 0.168) / 0.3 = 0.832 / 0.3 = 2.77
Speedup: 2.77 / (4 × 0.25 + 1) = 2.77 / 2.0 = 1.39×

The denominator factor is γc + 1 = 4·0.25 + 1 = 2.0: the early-exit draft, costing a quarter of the model per token, eats part of the benefit. Exiting higher up helps: with k = 4 (c = 0.125), the denominator is 1.5 and the speedup 2.77/1.5 = 1.85×. But exiting higher up normally lowers α, so there is real tension between a small k (cheap) and a high α (accurate).

An honest comparison with an external draft

Put a tiny, well-distilled external draft alongside: c = 0.1 and α = 0.78 (what an EAGLE-style draft can deliver), same γ = 4.

Tokens/step: (1 − 0.78⁵)/(1 − 0.78) = (1 − 0.289)/0.22 = 0.711/0.22 = 3.23
Speedup: 3.23 / (4 × 0.1 + 1) = 3.23 / 1.4 = 2.31×

Configuration	c	α	tokens/step	speedup	extra VRAM	pieces to maintain
Self-spec early-exit (k=8)	0.25	0.70	2.77	1.39×	0	0
Self-spec early-exit (k=4)	0.125	0.65	2.50	1.67×	0	0
Distilled external draft	0.10	0.78	3.23	2.31×	yes (+model +KV)	1 extra model

The reading is exactly what you would expect, and it is worth not dressing it up: if you have a dedicated draft, trained against your target, and it fits in memory, its higher α and lower c give it more speedup. EAGLE-3 with a well-trained draft usually wins on raw speedup. Self-spec does not compete on raw speedup; it competes on total cost. Its winning columns are the two on the right: zero extra VRAM and zero pieces to maintain. Self-spec wins when:

there is no trained draft for your model (your own SLM, an unusual fine-tune),
the draft does not fit (a 4090 already full, a long context that needs the KV),
you are on device (phone, NUC, edge), where a second model and its KV simply do not go in.

It is the same pattern as with MTP in the previous post: sometimes the best draft is the one you do not have to load.

Why it fits small models and device precisely

The regime where self-spec shines is low concurrency, memory-bandwidth-bound, with a tight memory budget, exactly that of an SLM on a single GPU or on device (the reason for the regime is in the inverted roofline). Three reasons:

Zero extra memory is decisive where there is none to spare. On an RTX 4090 (24 GB, Ada Lovelace) serving a quantised 7B–8B with a long context, every GB counts. Self-spec asks for none: it reuses weights and KV. An external draft, however small, forces you to trim context or concurrency. On device the difference is binary: with self-spec you accelerate; with an external draft there is no room, full stop.
There is no second checkpoint to version. Operationally, an SLM at the edge deployed across hundreds of boxes becomes unsustainable if each one needs two synchronised models. A single binary that does draft and verify is far simpler to maintain.
It fits MoE on device. In a fine-grained MoE for device (see native architectures for device), the shallow draft activates even fewer experts, and the memory-bound regime persists even at medium batch, precisely what the work on SSD for on-device MoE (ACM WWW 2026) exploits.

The counterpoint, repeated so it does not get forgotten: on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8), where memory is not the bottleneck, a dedicated EAGLE-3 draft does fit and its higher α gives it more speedup. There, self-spec is plan B: you use it if the model is exotic and there is no trained draft, not because memory is tight.

Pitfalls

α depends enormously on k. Exiting too high up (small k) makes the draft cheap but sinks α; exiting too low down (k close to L) raises α but the draft costs almost a full forward pass and c → 1, killing the speedup. The optimum is empirical and model-specific. Distrust any speedup number that does not say at which k it was measured.
Training-free is not free in draft quality. SWIFT and CLaSp give lower α than LayerSkip precisely because the model’s intermediate layers were not trained to be good premature exits. The number that matters is α measured on your distribution, not the paper’s.
Sampling temperature and creative outputs lower α just as in classic speculative. At high T, the self-spec speedup erodes faster still because it starts from a lower α.
Large batch neutralises it just as it does classic speculative. As soon as decode turns compute-bound, the drafts stop being “almost free”. Self-spec is for low concurrency.

References

Elhoushi, M., et al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. 2024. https://arxiv.org/abs/2404.16710
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration. ICLR 2025. https://openreview.net/forum?id=EKJhH5D5wA
CLaSp: In-Context Layer Skip for Self-Speculative Decoding. 2025. https://arxiv.org/abs/2505.24196
ConfLayers: Confidence-Adaptive Layer Skipping for Self-Speculative Decoding. 2026. https://arxiv.org/abs/2604.14612
Self-Speculative Decoding for MoE on Device. ACM Web Conference 2026. https://doi.org/10.1145/3774904.3792218
Hugging Face blog. Faster Text Generation with Self-Speculative Decoding. https://huggingface.co/blog/layerskip
Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192

The roofline flips: why optimising small models is a different performance game

Tue, 09 Jun 2026 01:30:00 +0000

This post is the anchor of a mini-series on inference performance in small models (SLMs). Almost every optimisation post on the blog, KV cache, decode, quantisation, was written with a 70B in mind. Here I argue that when the model shrinks by an order of magnitude, the roofline changes regime and several of those intuitions flip. This is not a nuance: it is a different game.

TL;DR

Autoregressive decode in a large LLM is memory-bandwidth-bound: at every step you have to move all the model weights from HBM to the registers of the SMs, and that dominates over the arithmetic operations. The GPU spends its time waiting for bytes, not computing. That single sentence, that decode “waits on HBM”, is the root of half the optimisations on this blog. In a small model (SLM, say 0.5B–7B) the sentence stops being true in the simple form we used to tell it. At batch 1 you are still memory-bound with respect to the hardware, yes, but the forward pass is so cheap (moving 6 GB at 1 TB/s takes ~6 ms, not 70 ms) that the fixed per-step costs, kernel launches, Python scheduler overhead, the sampler, host↔device copies, the synchronize calls, stop being noise and start eating 20-30 % of the time. The bottleneck moves from HBM to orchestration. The concrete, quantitative consequences: (1) CUDA graphs and cutting scheduler overhead pay off more in SLMs than in large models; (2) weight quantisation gives less latency improvement at batch 1 in SLMs, because proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead; (3) batching has more headroom because you cross the ridge point late; (4) the KV cache can dominate relative memory. All of this comes out of a single model, the roofline, applied with numerical honesty.

The analogy: the pantry and the waiter

A kitchen with two very different services.

Tasting menu service, one enormous slow dish (the 70B LLM). Every dish involves heavy ingredients that the assistant has to fetch from the pantry at the back, several times, carrying boxes. The chef, by contrast, assembles the dish in a moment: the slow part is fetching the ingredients, not cooking them. If you want the service to go faster, you do not hire a more skilful chef: you widen the corridor to the pantry, or make each trip bring more boxes. The pantry is HBM; the trip is memory bandwidth; cooking is compute. The big dish is bound by the pantry.

Tapas service, tiny plates (the SLM). Now each tapa has two ingredients and is assembled in a second. The trip to the pantry per tapa is very short. But a cost appears that was negligible for the big dish: the waiter. For every tapa, the waiter has to walk to the kitchen, pick up the plate, carry it to the bar, walk back, take the order, call it out. That back and forth is fixed: it costs the same for one tapa as for the enormous dish. When the tapa is assembled in a second, the waiter, not the pantry, is the bottleneck. Shortening the corridor to the pantry (widening HBM, quantising the weights) barely improves the service any more; what improves it is the waiter chaining several orders together without going back to the kitchen each time (CUDA graphs) or serving several tables in one pass (batching).

The roofline is the tool that says, with numbers, at what point the waiter dominates over the pantry. That boundary is the ridge point, and the joke in the title is that in SLMs we cross the regime much earlier than large-model intuition led us to believe.

The bare mechanism: what the roofline says

The roofline model (Williams, Waterman and Patterson, 2009) starts from a single quantity: arithmetic intensity, which is how many operations you do per byte moved from memory.

$$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}} \quad [\text{FLOP/byte}]$$

The hardware has two ceilings: compute (peak FLOPS) and memory (peak bandwidth × AI). Achievable performance is the minimum of the two:

$$\text{Perf} = \min\big(\text{peak FLOPS},\; \text{BW} \times \text{AI}\big)$$

Where the two lines cross is the ridge point, the AI above which you stop being limited by memory and start being limited by compute:

$$\text{AI}_{\text{ridge}} = \frac{\text{peak FLOPS}}{\text{peak BW}}$$

If your kernel has an AI below the ridge, you are memory-bound (the GPU waits for bytes). Above it, compute-bound (the GPU computes flat out and memory is in surplus). What matters is that the ridge point is a property of the hardware, not of the model. Let us look at the numbers, approximate, and I flag them as such because marketing figures mix dense and sparse, different dtypes and unrealistic thermal conditions.

Generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Per GPU, ~989 TFLOPS BF16 dense (~1,979 TFLOPS FP8 dense; the figure with sparsity is double that and almost never applies to LLM inference). HBM3 ~3.35 TB/s. The ridge in BF16:

$$\text{AI}_{\text{ridge}}^{\text{H100,BF16}} \approx \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \ \text{FLOP/byte}$$

In FP8 the ridge rises to ~590 FLOP/byte (twice the FLOPS against the same BW). Careful: these are datasheet peak figures; in practice a real kernel rarely gets past 70-80 % of either ceiling.

RTX 4090 (24 GB, Ada Lovelace). ~330 TFLOPS FP16 with FP16 accumulation via tensor cores (the “660 TOPS” figure that circulates is with sparsity), and ~1 TB/s of GDDR6X. The ridge:

$$\text{AI}_{\text{ridge}}^{\text{4090,FP16}} \approx \frac{330 \times 10^{12}}{1.0 \times 10^{12}} \approx 330 \ \text{FLOP/byte}$$

Curiously, the same order as the H100 in BF16: the 4090 has less BW but also fewer FLOPS, and the ratio comes out similar. The ridge sits around 300 FLOP/byte in both cases. Hold on to that number.

And where does decode fall? In decode at batch 1, each weight is loaded once from HBM and used for a single multiply-accumulate (one token, one row of activation). The AI of the decode GEMM at batch 1 is on the order of AI ≈ 1-2 FLOP/byte (each weight byte takes part in ~2 FLOP). With batch B, the same weight loaded once serves B rows of activation, so the AI scales roughly linearly:

$$\text{AI}_{\text{decode}}(B) \approx 2B \ \text{FLOP/byte} \quad (\text{for the weight GEMM part})$$

You cross the ridge when 2B ≈ 300, that is B ≈ 150 as an order of magnitude (in practice earlier, because of attention and overheads, but that is the frame). The clean conclusion: decode at low batch is always deeply memory-bound, nowhere near the ridge. That is why we say “decode waits on HBM”, and why quantising weights (moving fewer bytes) speeds up decode in a large model almost linearly. So far, this is the standard story from the large-model posts.

The point of the title: why it flips in SLMs

The classic roofline has a blind spot that does not matter in large models and is everything in small ones: it only models the work inside the kernel. It assumes the only time is bytes/BW or FLOPs/FLOPS. But a real decode step is not just the GEMM. It is a sequence of dozens of kernels (QKV projections, attention, the two MLP layers, normalisations, residuals, the logits head, sampling) and, around each one, there is a fixed orchestration cost:

Kernel launches (kernel launch): each cudaLaunchKernel costs on the order of 5-10 µs of CPU/driver overhead, regardless of kernel size. A decode forward with ~30-60 kernels launched sequentially drags ~0.3-0.6 ms just in launching.
Python scheduler overhead: the vLLM scheduler loop prepares metadata, decides which requests enter the step, builds the input tensors. In pure Python this is hundreds of µs to a couple of ms per step, especially at low concurrency where it is not amortised.
Sampling and post-processing: applying temperature, top-p, penalties, the argmax/multinomial, copying the token back. Another block of hundreds of µs.
Synchronisations and host↔device copies: every synchronize or small copy adds latency that is neither FLOPs nor HBM bytes.

Let us call the sum of all of that T_fixed, the per-step cost independent of model size, on the order of 1-3 ms in an unoptimised Python stack. The real time of a step is now:

$$T_{\text{step}} \approx \underbrace{\frac{\text{weight bytes}}{\text{BW}}}_{T_{\text{HBM}} \text{ (memory-bound)}} + \; T_{\text{fixed}}$$

In a 70B BF16, moving ~140 GB at 3.35 TB/s takes ~42 ms of T_HBM. Against that, a T_fixed of 1-3 ms is noise (2-7 %). The classic roofline is right: the model is memory-bound, full stop. But in a 3B BF16, T_HBM drops to a few ms, and suddenly T_fixed is the same order as T_HBM. The bottleneck stops being the pantry and becomes the waiter. This is the flip in the title, and four counterintuitive consequences follow from it:

(a) At batch 1 you are still memory-bound with respect to the hardware. The AI has not changed: it is still ~2 FLOP/byte, below the ridge. Anyone reading only the roofline will conclude “memory-bound, quantise the weights”. That is true but incomplete: the roofline does not see T_fixed.

(b) Fixed costs become a huge fraction of the step. This is the central point. In the 70B, T_fixed / T_step ≈ 5 %. In the 3B it can be 20-30 %. The effective bottleneck of the 3B is half HBM, half orchestration.

(c) That is why CUDA graphs and cutting scheduler overhead pay off MORE in SLMs. A CUDA graph captures the whole kernel sequence of the step and relaunches it with a single cudaGraphLaunch, removing almost all of the per-kernel launch overhead and much of the per-iteration Python scheduler work. In the 70B, shaving 0.5 ms off a 42 ms step is a +1 % you barely notice. In the 3B, shaving those same 0.5 ms off a ~7 ms step is +7 %, and if you take away nearly all of T_fixed you can gain 20-30 %. The same optimisation, a different prize, because the denominator changed.

(d) Weight quantisation gives LESS latency improvement at batch 1 in SLMs. This is the most counterintuitive one. In the 70B, T_HBM is almost the whole step; going from BF16 to INT4 quadruples the effective weight bandwidth and almost quadruples decode speed. In the 3B, T_HBM is only part of the step (the rest is T_fixed + attention + KV). By Amdahl’s law, if the weights are 60 % of the step and you speed them up 4×, the total step improves only 1/(0.4 + 0.6/4) = 1.8×, not 4×. And proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead. Aggressive quantisation in SLMs does help, yes, but not through raw latency at batch 1, where it gives diminishing returns, rather through capacity and concurrency (we will see that at the end).

(e) The KV cache can dominate relative memory. With 6 GB of weights (3B BF16), a single long-context session can approach that order of magnitude in KV cache. In a 70B (140 GB of weights) the KV is proportionally small until high concurrency. In SLMs the VRAM balance tips towards KV much earlier (the detail is in KV cache), and that changes which memory optimisation is the lever.

The maths that matter: a 3B on a 4090

Let us do the whole calculation, which is where the flip shows up without rhetoric.

Model: 3B parameters, BF16 → 2 bytes/param → ~6 GB of weights. Hardware: RTX 4090, BW ≈ 1 TB/s.

Memory-bound ceiling for decode (batch 1). Each token requires loading the 6 GB once:

$$T_{\text{HBM}} = \frac{6 \times 10^{9} \ \text{bytes}}{1 \times 10^{12} \ \text{bytes/s}} = 6 \times 10^{-3}\ \text{s} = 6\ \text{ms/token}$$ $$\text{Ceiling} = \frac{1}{6\ \text{ms}} \approx 166\ \text{tok/s}$$

That is the theoretical memory-bound ceiling: 166 tok/s, assuming that moving the weights is the only cost. The classic roofline would stop here and say “166 tok/s, go get more BW or quantise”.

Now the fixed overhead. Let us put T_fixed ≈ 2 ms/step (a reasonable value for a Python scheduler + ~40 kernels launched + sampling, without CUDA graphs). The real step:

$$T_{\text{step}} = T_{\text{HBM}} + T_{\text{fixed}} = 6 + 2 = 8\ \text{ms} \;\Rightarrow\; \frac{1}{8\ \text{ms}} = 125\ \text{tok/s}$$

The overhead has eaten 41 tok/s out of the theoretical 166: T_fixed is 25 % of the step (2 of 8 ms). Compare with the 70B: T_HBM ≈ 42 ms, T_step ≈ 44 ms, T_fixed is 4.5 %. Same absolute overhead, relative impact 5-6× larger in the SLM.

What happens if you apply CUDA graphs and take away, say, 1.5 of the 2 ms of T_fixed:

$$T_{\text{step}}^{\text{graphs}} = 6 + 0.5 = 6.5\ \text{ms} \;\Rightarrow\; 154\ \text{tok/s}$$

From 125 to 154 tok/s: +23 % from orchestration alone, without touching the model or the memory hardware. In the 70B the same intervention would have gone from 44 to 42.5 ms, +3.5 %. There it is, in two numbers: “a different game”.

What happens if you quantise the weights to INT4 (1.5 GB instead of 6 GB), with T_fixed still at 2 ms:

$$T_{\text{HBM}}^{\text{INT4}} = \frac{1.5 \times 10^{9}}{1 \times 10^{12}} = 1.5\ \text{ms};\quad T_{\text{step}} = 1.5 + 2 = 3.5\ \text{ms} \;\Rightarrow\; 285\ \text{tok/s}$$

The 4× weight quantisation did not give 4× in latency: it went from 125 to 285 tok/s, a 2.3×, because the 2 ms of T_fixed now dominates (it is 57 % of the step). In the 70B, quantising to INT4 gives almost the full 4× because T_fixed is still noise. The same quantisation delivers twice the speedup on the large model as on the small one, at batch 1. And if you also apply CUDA graphs on top of INT4 (T_fixed → 0.5 ms): 1.5 + 0.5 = 2 ms → 500 tok/s. The order of the optimisations matters: in SLMs, attacking T_fixed first unlocks the rest.

Configuration (3B, 4090, batch 1)	T_HBM	T_fixed	T_step	tok/s	vs. baseline
BF16, no graphs (baseline)	6.0 ms	2.0 ms	8.0 ms	125	1.00×
BF16 + CUDA graphs	6.0 ms	0.5 ms	6.5 ms	154	1.23×
INT4, no graphs	1.5 ms	2.0 ms	3.5 ms	285	2.28×
INT4 + CUDA graphs	1.5 ms	0.5 ms	2.0 ms	500	4.00×

(Illustrative figures with T_fixed rounded; the point is the pattern, not the decimal. The real T_fixed depends on the stack, the PyTorch/CUDA version and whether there is tensor parallelism. Measure it in your own setup before believing any row.)

Implications per optimisation

With the model in hand, the blog’s levers reorder themselves once the regime changes.

Batching: far more headroom in SLMs. Remember that you cross the ridge at B ≈ ridge/2 ≈ 150 as an order of magnitude. In a large model, VRAM runs out long before you saturate compute (weights + KV do not let you reach batch 150). In an SLM the weights take up little space, so you can fit large batches in VRAM and stay memory-bound over a much wider range: the T_HBM of the weights is amortised across the B requests (you load them once, they serve B), so aggregate throughput per GPU rises almost linearly with B until very high values. This is exactly the opposite of the 70B’s fear of saturating compute. In SLMs, batching is the throughput lever par excellence because you saturate compute late; the batch grid search in vLLM has a much wider plateau of good behaviour. Careful: batching improves throughput, not per-request latency; for single-stream latency the prize is in T_fixed.

Speculative decoding: a different crossover point. Speculative wins when verifying γ tokens is “almost free” because you are memory-bound. In an SLM the target is already cheap, so the draft has to be tiny for c = T_draft/T_target to stay small, and the draft’s own T_fixed (launching its kernels) bites harder. The crossover to compute-bound with batching also arrives earlier in absolute terms of tok/s served. The variant that fits best here avoids a separate draft: self-speculative / early-exit reuses early layers of the model itself and saves the T_fixed of orchestrating two models.

Quantisation: it helps for capacity, not for latency at batch 1. As the table showed, INT4 in an SLM at batch 1 gives diminishing returns in latency. Its real prize in SLMs is capacity: weights 4× smaller free VRAM for more KV cache → more concurrency, and it is at high concurrency (aggregate throughput) that saving bytes pays off again. Aggressive sub-4-bit and ternary quantisation takes this to the extreme: in SLMs it makes sense above all for fitting more sessions per GPU, not for lowering the latency of a single one. And it is worth remembering (see quantisation) that at batch 1 the dequantize adds compute work which, in a regime already grazed by T_fixed, is not always free.

Architecture: fine-grained MoE changes which bytes you move. A fine-grained device-native MoE activates few parameters per token, so T_HBM drops relative to a dense model of the same total size, but the T_fixed fraction rises even further, and the router adds its own fixed overhead. It is the SLM regime taken to its limit: almost the whole game is played in orchestration.

Scheduler and CUDA graphs first. The operational conclusion, inverted with respect to the large-model posts: in SLMs, before touching the model, kill T_fixed. CUDA graphs (see SMs, streams and graphs), a vLLM scheduler with its Python part minimised or compiled, and kernel persistence are the first-order levers. In a 70B they would be marginal polish; in a 3B they are half the available speedup.

Applied to on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace). This is the scenario where the flip is most visible, because the 4090 has ~1 TB/s (a third of the H100) but T_fixed is the same in absolute terms. A 3B BF16 without CUDA graphs leaves ~125 tok/s on the table when the memory-bound ceiling is 166; enabling graphs and cleaning up the scheduler recovers most of that. The 4090 has plenty of VRAM room for an SLM, so the bottleneck is almost never total memory but orchestration and, at high concurrency, the KV cache. Rule of thumb: on a 4090 with an SLM, profile the per-step overhead first (Nsight Systems on the gap between kernels) before you quantise.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). The H100 has 3.35 TB/s, so the T_HBM of an SLM is even smaller (a 3B FP8 is ~3 GB → ~0.9 ms) and T_fixed dominates even earlier: a badly orchestrated SLM on an H100 can spend more time in the Python scheduler than moving weights. Serving a single single-stream SLM on an H100 is close to a waste; the right mode is aggressive batching (you saturate compute late, so you push large batches through and per-GPU throughput takes off) or multiplexing many SLMs/sessions per GPU via MPS/MIG. This connects with capacity planning: for SLMs the capacity calculation is governed by concurrency and KV cache, not by the weights. And with the dilemma of one big versus N small: replicating SLMs makes sense precisely because each replica saturates compute late and TP brings nothing (the model already fits; TP would only add communication T_fixed).

What we have not covered

The exact measured T_fixed, kernel by kernel, with Nsight Systems: how much is launch, how much scheduler, how much sampling. That is the content of the next post in the series.
torch.compile / partial captures: alternatives and complements to CUDA graphs when there is dynamic control flow.
The prefill regime in SLMs: prefill is compute-bound even in small models (it processes many tokens at once, high AI), so its roofline is the opposite of decode’s; see prefill.
Attention and KV as the second term of T_HBM: here we have folded them in implicitly; the fine-grained breakdown of attention (which scales with sequence length, not with the weights) deserves its own treatment.

References

Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009. https://doi.org/10.1145/1498765.1498785
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv:2503.08311, 2025. https://arxiv.org/abs/2503.08311
Databricks. LLM Inference Performance Engineering: Best Practices. https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices
NVIDIA. NVIDIA H100 Tensor Core GPU Datasheet. https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet
NVIDIA. GeForce RTX 4090 — product specifications (Ada Lovelace tensor core figures; treat as approximate, they mix dense/sparse).
Yuan, Z. et al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363, 2024 — application of the roofline specifically to LLM inference.

Slm on lo0 — Tech Blog

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

TL;DR

The end-to-end flow

Phase 0 — Environment and versions

Phase 1 — Preparing the dataset

Phase 2 — The training script

BitsAndBytesConfig — how the base is quantised

LoraConfig — the shape of the adapter

SFTConfig — the memory budget and the loop

Phase 3 — Launch and monitor

Phase 4 — Validating the adapter

Phase 5 — Versioning the adapter as an artefact

Phase 6 — Serving in multi-LoRA with vLLM

Serving multi-adapter vs merging per task

Checklist of operational gotchas

Applied to on-premise infrastructure

See also

References

QLoRA and multi-LoRA at the limit on small models

TL;DR

The analogy: the frozen guitar and the swappable pedalboard

The bare mechanism: LoRA, and why you can train on a 4-bit base

NF4: why a new format instead of INT4

Double quantisation and paged optimizers

“Aggressive” training: very low rank and QA-LoRA

The maths that matters

The fit with small models and sovereignty

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

Aggressive RAG in small models: trading parameters for retrieval

TL;DR

The analogy: the open-book exam

The capacity argument: how many facts fit in the weights

The central tension: retrieving more is not putting in more

(a) SLMs make worse use of long context

(b) Prefill grows with the context and is compute-bound

The maths of prefill

Numerical example: TTFT before and after compressing, RTX 4090

The five levers to resolve the tension

1. Aggressive reranking: precision over recall

2. Context compression: distilling the crib sheet

3. Prefix caching of stable context

4. Semantic caching of answers

5. Structured output and function calling: lean on tools, not on memory

The complete pipeline

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

Aggressive quantisation (state of the art): from 4-bit to ternary

TL;DR

The analogy: the JPEG that cannot be compressed any further

The map of the frontier, bit by bit

Why scalar PTQ collapses below 4 bits

1. Vector quantisation: dictionaries instead of scales

2. Incoherence processing: whitening the matrix

3. Coding with memory: trellis

Native ternary: BitNet b1.58

What changes is not just the memory, it is the arithmetic

QAT as the bridge between PTQ and native

The maths that matter: footprint and how many fit

How many 8B models fit on an RTX 4090?

The conceptual curve: perplexity vs bits

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Implications for on-premise inference

See also

References

Test-time quantization: quantising on the fly with no calibration dataset

TL;DR

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

The problem TTQ solves: fixed calibration ages with the traffic

The TTQ mechanism: measure the real activations and scale on the fly

`BitsAndBytesConfig` — how the base is quantised

`LoraConfig` — the shape of the adapter

`SFTConfig` — the memory budget and the loop

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

The maths: same framework, different `c`