Aggressive quantisation (state of the art): from 4-bit to ternary

This post is the direct continuation of Quantization for LLM inference, which covers the “solved” regime (FP8, INT4 with GPTQ/AWQ). Read it first: here I assume the scale+zero-point maths, what GPTQ and AWQ do, and the PTQ/QAT distinction. What we add is the sub-4-bit frontier, where scalar post-hoc quantisation stops working and you have to change tools.

TL;DR

There is a sharp dividing line around 4 bits. Above it, quantising is a solved problem: INT8 is indistinguishable from BF16, and INT4 with a good method (AWQ, GPTQ) loses 1-2 MMLU points and little else. The method is still the usual one, take each weight, scale it, round it to a short integer, and it works. Below 4 bits, that method collapses: at 2 bits, naive scalar quantisation can double perplexity. The reason is geometric, each weight has only 4 possible values and the rounding error stops being negligible, and the way out is not “round better”, it is to change the representation. State-of-the-art 2-bit methods (AQLM, QuIP#, QTIP) stop quantising individual weights and quantise vectors of weights against dictionaries (codebooks), and they “whiten” the weight matrix to spread its energy and flatten outliers (incoherence processing). Ternary is something else again: BitNet b1.58, with weights in {-1, 0, +1} (~1.58 bits), is not PTQ, it is a model trained natively under that constraint, and it changes the arithmetic of the matmul from multiplications to additions and subtractions, hitting the compute ceiling and the memory ceiling at once. The mental rule: at ≥4-bit you compress the photo; below 4-bit you have to repaint it.

The analogy: the JPEG that cannot be compressed any further

In the quantization post we used the JPEG with an edge detector to explain INT4. The analogy still holds here, but it has to be pushed to its limit.

A JPEG at quality factor 90 is indistinguishable from the original. At quality 60 you notice it a little, but it is still “the same photo”. At quality 30 the 8×8 blocks appear, the haloes around edges, the banding in gradients. At quality 10 the image is destroyed: you can tell there was a face, but the details have disappeared under the artefacts. And here is the key: no JPEG encoder exists that can compress to quality 10 without those artefacts, because the JPEG algorithm (blockwise DCT + coefficient quantisation) has an information floor below which its own mechanism introduces the noise.

What do you do if you need the photo at that file size and you need it to still look good? You do not compress the original harder. You repaint the photo knowing in advance that it is going to live compressed: an illustrator redraws it with clean lines, a reduced palette, no subtle gradients, an image designed to survive brutal compression. The result at “10 KB” looks infinitely better than the original JPEG crushed to 10 KB, because it is not the same process: one destroys existing information, the other generates new information already adapted to the constraint.

That is exactly the frontier of this post:

• Scalar PTQ (≥4-bit) = compressing the JPEG. Up to a certain ratio, it is still the same photo.
• State-of-the-art vector PTQ (2-bit: AQLM, QuIP#, QTIP) = a far more sophisticated image codec (dictionaries, decorrelating transforms) that stretches the compressible ratio a bit further before the collapse.
• Native ternary (BitNet b1.58) = repainting the photo. You are not compressing an existing BF16 model; you are training a new one that is born ternary.

The map of the frontier, bit by bit

Quantising a model is deciding how many distinct values each weight can take. With b bits per weight there are 2^b possible values. The central question is: past what b is the number of values so small that rounding destroys the model?

The three transitions that matter:

8 → 4 bits: nothing breaks. With 16 levels per weight and one scale per block of 128, the rounding error is small relative to the dynamics of the weights. GPTQ compensates for the error by propagating it to the neighbouring weights; AWQ protects the ~1 % of salient channels. The model barely notices. This is in the previous post.

4 → 2 bits: the elbow. Two things happen at once here. First, with only 4 levels, the scalar quantiser can no longer represent the weight distribution, which is roughly Gaussian with long tails, without a proportionally enormous rounding error. Second, and more subtly: the quantisation error stops being “small noise that the model absorbs” and becomes structured, systematically biasing the activations. Naive scalar PTQ at 2 bits on a Llama 8B typically doubles perplexity or worse. That is the elbow of the curve.

2 → 1.58 bits: a change of nature. You do not cross this with a better compression method. You cross it by training the model with the constraint from the start. It is a discontinuity: to the left you are doing PTQ, to the right you are doing training.

Why scalar PTQ collapses below 4 bits

The scalar quantiser has a fundamental limitation: it quantises each weight separately, ignoring the fact that the weights of a row or column are correlated and that the error of one could be compensated by another. At 4 bits this matters little; at 2 bits it is lethal. There are three possible attacks, and the state-of-the-art methods use all three.

1. Vector quantisation: dictionaries instead of scales

Instead of mapping each weight to one of 4 values, group the weights into vectors (say, of 8 weights) and map each vector to the nearest entry of a learned dictionary (codebook). If the dictionary has 256 entries, encoding a vector of 8 weights costs 8 bits (the index) → 1 bit/weight, but every “reconstructed value” is a point in an 8-dimensional space chosen to minimise the error over the real distribution of the weights.

The advantage is information-theoretic: a vector dictionary can place its reconstruction points where the weights actually are (in clusters), whereas the scalar quantiser is forced to put its 4 levels on a regular grid, wasting resolution on empty regions. It is the difference between a road map with a uniform grid and one that puts more detail where the cities are.

AQLM (Additive Quantization of Language Models, arXiv:2401.06118) takes this to the extreme with additive quantisation: each weight vector is reconstructed as the sum of several codes from several dictionaries (multi-codebook). It is more expressive than a single dictionary because the number of combinations is the product of the sizes, not the sum. AQLM was one of the first methods to make 2-bit “usable” (not collapsed) on large models, at the cost of an expensive calibration process and specialised inference kernels.

2. Incoherence processing: whitening the matrix

The second attack targets the outliers. The weight matrices of a transformer have a few entries (and a few channels) with a magnitude much larger than the rest. Those outliers dominate the quantiser’s range: if you have to represent a weight of magnitude 8 while the rest are of magnitude 0.5, your scale stretches to cover the 8 and you waste almost all the resolution.

Incoherence processing (the core idea of QuIP and QuIP#) attacks this by multiplying the weight matrix W by random orthogonal matrices on the left and on the right: W' = U W V^T. Because U and V are orthogonal, the operation is invertible and the maths of the product can be undone at inference time by absorbing it into the neighbouring layers (just as AWQ absorbs its scales). But the rotation spreads the energy: an “incoherent” matrix has its values distributed almost uniformly, with no concentrated outliers, because mixing coordinates with a random rotation flattens the distribution (it is, in essence, the central limit theorem acting on linear combinations). A matrix without outliers quantises far better at 2 bits. It is the equivalent of “whitening” a signal before digitising it.

QuIP# (arXiv:2402.04396) combines incoherence processing with E8 lattice codes: rather than an arbitrary dictionary, it uses the E8 lattice (an optimal sphere packing in 8 dimensions, the best known). Quantising vectors of 8 weights against the E8 lattice gives the smallest reconstruction error possible for a given bit density, because E8 is literally the most efficient way to place points in 8D. It is classical coding theory applied to LLM weights.

3. Coding with memory: trellis

QTIP (arXiv:2406.11235) adds the third attack: trellis-coded quantisation. Instead of quantising each vector independently, it models the sequence of weights as a path through a trellis (the same structure as the convolutional codes of telecommunications) and picks the optimal code sequence with the Viterbi algorithm. The intuition: introducing memory between successive quantisations allows correlated errors that cancel out, instead of independent errors that accumulate. QTIP, on top of incoherence processing, improves on QuIP# in quality at 2-3 bits while keeping fast inference kernels.

The idea common to all three: stop quantising scalars and start quantising vectors with dictionaries, and decorrelate the matrix before doing so. None of them is “rounding better”; all three change the representation at the root. That is why, below 4 bits, a flag in vLLM is no longer enough: you need co-design of the quantisation method and the inference kernel.

Native ternary: BitNet b1.58

Here we change continents. Everything above is PTQ: it starts from a trained BF16 model and compresses it. BitNet’s ternary compresses nothing.

BitNet b1.58 (arXiv:2402.17764) trains a transformer from scratch where every weight is constrained to {-1, 0, +1} throughout training. Three values ⇒ log₂(3) ≈ 1.58 bits/weight. Quantisation is not a later step: the linear layers (BitLinear) quantise their weights to ternary in the forward pass of every training step, and gradients flow through a straight-through estimator. The model learns to work with ternary weights. This is QAT taken to the extreme: not a short robustness fine-tune, but the constraint present from the first token of training.

That difference is what dodges the elbow of the curve. PTQ at 2 bits tries to find the best ternary or quaternary approximation of a model that was trained expecting full precision, and that model has “fragile” weights that depend on nuances 2 bits cannot capture. BitNet, by contrast, never had those nuances: its weights were born ternary, so the network distributed its representational capacity in a way compatible with the constraint. It is repainting the photo instead of compressing it.

What changes is not just the memory, it is the arithmetic

The most underrated point about BitNet: with weights in {-1, 0, +1}, multiplication disappears from the matmul. Multiplying an activation x by a ternary weight w is trivial: if w = +1 you add x, if w = -1 you subtract x, if w = 0 you do nothing. The dominant operation of a transformer, the matrix-vector product, goes from a sea of floating point multiply-accumulate (MAC) to integer additions and subtractions.

This matters because it connects to the roofline. As explained in The inverted roofline of small models, LLM inference has two ceilings: memory (HBM bandwidth for loading weights) and compute (tensor core FLOPs). Normal quantisation (INT4, FP8) attacks only the memory ceiling: the weight takes up less space, but to multiply it you dequantise to FP16 and do the same multiplication as always. Ternary attacks both ceilings at once: the weight occupies 1.58 bits (memory) and the operation is an addition instead of a multiplication (compute). That is why BitNet needs its own kernels, bitnet.cpp, that execute the ternary matmul without ever going through FP16; a kernel that dequantised to FP16 in order to multiply would throw away half the advantage.

The honest trade-off: BitNet b1.58 is training from scratch. You cannot “convert your Llama 8B to BitNet”. If you want ternary, you train (or use) a natively ternary model, with everything that implies in pre-training cost and weight availability. Today it is a research line with published models at modest scales, not a drop-in replacement for your current serving.

QAT as the bridge between PTQ and native

Between “compressing post-hoc” (PTQ) and “training natively ternary” (BitNet) there is a middle point: QAT (Quantization-Aware Training). You take an already trained model and do a short fine-tune with the quantisation operations inside the loop, so that it learns to be robust to low bit counts without paying for a full pre-training.

Gemma 3 publishes official QAT variants for exactly this: models that, after the QAT fine-tune, hold up at INT4 with far less quality loss than pure PTQ on the same model. The cost is a training cost (GPU hours to days on an existing model), not an inference cost. For INT4 with QAT you recover almost all the quality; for 2-bit, QAT helps but the terrain is still difficult; for ternary, QAT stops being a “short fine-tune” and becomes native training (BitNet).

The decision hierarchy:

• PTQ = default at ≥4 bits. Minutes to hours, without touching training weights. It covers 90 % of production.
• QAT = when PTQ loses too much and the difference matters. Low bits (2-3), or sensitive models. You pay for a fine-tune.
• Native (ternary) = when you want to go below 2 bits and change the arithmetic. You pay for pre-training. It only makes sense if you control the model from its creation.

The maths that matter: footprint and how many fit

The weight footprint is direct: bytes = (bits/param / 8) × N, with N the number of parameters. For an 8B model:

(Real ternary takes up slightly more than 1.58 bits/param because you have to pack 5 ternary values into 8 bits, 5 × log₂(3) ≈ 7.92 bits, and because norms and embeddings usually stay at higher precision. The ~1.6 GB figure for 8B is the right order of magnitude.)

How many 8B models fit on an RTX 4090?

An RTX 4090 (24 GB, Ada Lovelace) has 24 GB. Reserving ~4 GB for KV cache and activations leaves 20 GB for weights. How many distinct 8B models fit loaded at the same time:

The arithmetic is seductive, from 1 to 12 models on the same card, but it should be read sceptically. Jumping from INT4 (5 models, almost lossless) to 2-bit (10 models) doubles the capacity, but only if you use a state-of-the-art method and accept 4-8 MMLU points. And the jump from 2-bit to ternary (10 → 12) is marginal in memory terms: ternary is not justified by footprint against a state-of-the-art 2-bit, it is justified by the arithmetic (the compute ceiling) and because being native it avoids the quality elbow. If your only metric is “how many GB it takes”, state-of-the-art 2-bit already gives you nearly everything. Ternary is for when you also want the compute saving.

The conceptual curve: perplexity vs bits

Three readings of the curve. One: to the right of 4 bits, the three branches are almost stuck together and almost flat, the solved regime. Two: the red branch (naive scalar PTQ) has a brutal elbow between 3 and 2 bits; that is where it doubles perplexity. The blue branch (state-of-the-art vector PTQ) flattens that elbow, it does not remove it, but it makes it tolerable down to 2 bits. Three: the green point of native ternary is on neither of the two PTQ curves, because it is not obtained by compressing: it is obtained by training, and that is why it can fall below the elbow without paying the quality price that any PTQ pays at that bit density. It is the difference between the crushed JPEG and the repainted photo.

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Three warnings for reading the aggressive-quantisation literature:

“1-bit lossless” almost always comes with small print. Pure binary {-1, +1} (1 bit) loses the ability to represent zero, which matters in transformers (many weights are effectively null). That is why the true state of the art at low density is ternary (1.58 bits), not binary: the zero is worth its extra 0.58 of a bit. When a paper announces “1-bit”, it pays to check whether (a) it really is 1 bit or 1.58 rounded down in the headline, (b) “lossless” is measured in WikiText perplexity (easy) or on reasoning benchmarks (where the collapse shows up), and (c) it compares against a baseline of the same effective size or against a much larger model in order to inflate the advantage.

Flat perplexity ≠ preserved quality. Perplexity on a generic corpus is the metric most forgiving of aggressive quantisation. A 2-bit model can have a perplexity almost identical to BF16 and at the same time drop 10 points on GSM8K or on a code benchmark, because multi-step reasoning amplifies errors that average perplexity does not see. Distrust any sub-4-bit claim that only reports perplexity. As we already said in the quantization post, the loss has to be measured on the target task.

Hardware comparability. The “X times faster” numbers for ternary only apply with the specialised kernels (bitnet.cpp) and on hardware where addition/subtraction arithmetic genuinely wins. On a GPU with tensor cores designed for FP16/FP8, a naive ternary kernel can be slower than a well-optimised INT4, because it wastes the silicon. The ternary advantage is real, but it is a co-design advantage (model + kernel + sometimes hardware), not a flag you switch on over your current stack. Any benchmark that does not specify the exact kernel and hardware is noise.

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace): the practical regime today is still INT4 AWQ for 7-14B models, solved, almost lossless, natively supported. State-of-the-art 2-bit (AQLM/QuIP#/QTIP) is viable and lets you load larger models or more models at once, but it demands the specific kernels of each method and an expensive calibration, and it pays in quality. It makes sense when VRAM is the bottleneck and you accept the trade-off; not as a default. Ternary on a 4090 is experimental: without tensor cores designed for ternary addition and subtraction, the compute advantage is diluted, although the memory saving remains.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8): here the default is FP8 (almost indistinguishable quality, native throughput) or INT4 AWQ for models that do not fit in FP8. State-of-the-art sub-4-bit is for serving enormous models (200B+) when neither FP8 nor INT4 fits with the KV cache headroom you want, at the cost of quality and kernel complexity. Native ternary, today, is a research object rather than a production one: its promise, hitting both roofline ceilings, is greater on CPU/edge (where there are no FP8 tensor cores to exploit) than on an H100 cluster, which already has dedicated FP8 hardware.

The rule of thumb, June 2026: ≥4-bit is settled engineering; state-of-the-art 2-bit is a real lever but with a cost in method and in quality; ternary is an architectural bet, not a deployment tweak.

See also

• Quantization for LLM inference: FP8, INT4, GGUF — the indispensable base: the scale+zero-point maths, GPTQ/AWQ and PTQ vs QAT, all taken for granted here; this post is its continuation towards the sub-4-bit frontier.
• FP8 end to end: weights, KV and quality — the other end of the spectrum, the solved datacenter regime where quantisation costs almost no quality.
• Knowledge distillation — the complementary lever: distilling reduces parameters, quantising reduces bits per parameter; at 2-bit they are usually combined to reach the target footprint.
• Pruning LLM models — sparsity and aggressive quantisation are orthogonal and they compound: 50 % sparse + 2-bit is another route to the same footprint as ternary.
• KV cache: the working memory — the ~4 GB we reserve for KV in the 4090 arithmetic come from here; quantising the cache is the other half of the memory budget.
• The inverted roofline of small models — why ternary is special: it attacks the memory ceiling and the compute ceiling at once, while INT4/FP8 only touch the memory one.
• Test-time quantization on the fly — quantising dynamically at inference time against the static, calibrated quantisation that AQLM/QuIP#/QTIP describe.
• Device-native architectures + fine-grained MoE — on-device Q4 as the starting point of which sub-4-bit and ternary are the next frontier for edge.
• QLoRA and aggressive multi-LoRA on SLMs — fine-tuning on an already quantised base; the limit of how far you can compress the base before the adapter can no longer recover the quality.

References

• Ma, S. et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58). https://arxiv.org/abs/2402.17764
• Egiazarian, V., Panferov, A., Kuznedelev, D. et al. Extreme Compression of Large Language Models via Additive Quantization (AQLM). https://arxiv.org/abs/2401.06118
• Tseng, A., Chee, J., Sun, Q., Kuleshov, V., De Sa, C. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. https://arxiv.org/abs/2402.04396
• Tseng, A., Sun, Q., Hou, D., De Sa, C. QTIP: Quantization with Trellises and Incoherence Processing. https://arxiv.org/abs/2406.11235
• Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
• Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
• Google DeepMind. Gemma 3 QAT (Quantization-Aware Training) models — official blog: https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/
• Microsoft. bitnet.cpp — 1-bit ternary inference kernels: https://github.com/microsoft/BitNet