Awq on lo0 — Tech Blog

Test-time quantization: quantising on the fly with no calibration dataset

Tue, 09 Jun 2026 02:00:00 +0000

This post is the natural continuation of Quantization for LLM inference, which is worth reading first: that is where GPTQ, AWQ, scale + zero-point and why activation outliers are the central problem all live. Here we are not discussing how many bits to use, but when and with what information the scales are computed: offline against a corpus (PTQ) or on the fly against real traffic (TTQ).

TL;DR

Activation-aware quantisation (AWQ, SmoothQuant) decides which channels to protect by measuring the magnitude of the activations over a calibration dataset in an offline pass, before deployment. The implicit assumption is that this corpus represents future traffic. But activation outliers, the channels with 10-100× the median magnitude that dominate the quantisation error, depend on the input: they change with the domain, the language and the client’s distribution. When real traffic drifts away from the calibration, the fixed scales stop being optimal and quality drops. Test-time quantization (TTQ) removes the corpus and the offline pass: it derives the activation-aware scales at inference time, from the activations that are actually observed, per token or per batch. The trade-off is honest and not minor: it introduces runtime overhead, computing statistics, detecting outliers, recomputing scales at every step, which competes directly with the saving from quantising. On small models that overhead weighs proportionally more, because the forward is short and the fixed per-step costs dominate (the framing is in the inverted roofline for SLMs). TTQ is orthogonal to the format: it is not a competitor to INT4 or FP8, it is a different way of deriving s. It pays off when there is no calibration pipeline, when the traffic distribution is shifting or unknown, and in multitenant settings where no representative corpus exists.

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

A clothes shop has two ways of dressing a customer.

The first is to sell off-the-peg sizes. At some point the factory measured an “average customer”, a mean mannequin built from a population sample, and cut the garments to those measurements. When a customer walks in, you hand them the size that comes closest. It is extremely fast: the garment is already sewn, it just gets handed over. The problem appears when the customer does not look like the average mannequin: if their shoulders are much broader than average, their particular outlier, the standard size pulls or leaves spare fabric, because it was cut to protect other areas. This is offline calibrated PTQ: AWQ measured the importance of each channel over a corpus and fixed the scales once and for all; fast at inference time, but blind to the specific customer.

The second is the tailor who takes measurements on the spot. When the customer walks in, the tailor gets out the tape, measures that customer, spots where their particular bulk is and adjusts the cut to their real anatomy. The result fits better, especially for customers who fall outside the mould. But every customer costs time: measuring, marking, deciding. This is TTQ: the scales are derived on the fly from the activations that the input actually generates.

The analogy holds up in three details:

The average mannequin = the calibration dataset. If the population walking into the shop resembles the mannequin, the sizes work; if not, they fail at the extremes.
Measuring every customer = computing activation statistics per token/batch. A better fit, but a fixed cost paid on every garment.
The broad shoulders = the outlier activation channels. They are precisely the areas where the fit matters and where the generic size gets it most wrong.

The tailor wins when the customers are varied or unknown. The tailor loses when you have a homogeneous population and a mannequin that represents it well: there, paying for measurement on every customer is a waste of time.

The problem TTQ solves: fixed calibration ages with the traffic

Let us recall from the quantization post what AWQ and SmoothQuant do exactly. They do not quantise all channels equally: they identify the ~1 % of channels whose activations have a large magnitude, the salient channels, and protect them by scaling them before quantising. To measure that importance they need to see activations, and they see them over a calibration dataset (128-512 samples, typically WikiText or a slice of the domain) in an offline pass prior to deployment.

The assumption is strong: that the activation distribution of the calibration corpus represents that of production traffic. Two reasons why that assumption breaks:

Activation outliers depend on the input. They are not a fixed property of the model the way weights are. The channel that is an outlier when processing C++ code may not be one when processing conversational Arabic or log JSON. The magnitude and position of the peaks change with the domain, the language and the input format.
Real traffic is rarely the corpus. You calibrate with English WikiText and the client sends you support tickets in Spanish with tables pasted in. The calibration protected the channels that WikiText activated, not the ones that real traffic activates. The scales are suboptimal exactly where the client lives.

The result is distribution-dependent degradation: the quantised model keeps its quality while the input resembles the calibration and loses it as the input drifts away. The most awkward case is multitenant: if you serve clients with different domains from the same quantised model, no single representative corpus exists; any fixed calibration favours some tenants and penalises others.

The TTQ mechanism: measure the real activations and scale on the fly

TTQ (arXiv:2603.19296, March 2026) proposes deriving activation-aware quantisation at inference time, with no offline pass and no calibration dataset. The idea, in its bare conceptual form:

Step 1 — Observe. When the activation tensor X arrives at a linear layer (per token or per batch), cheap statistics are computed over the channels: a measure of central tendency (median or mean magnitude) and one of dispersion per channel. This is the equivalent of AWQ looking at its corpus, but done over the activations that really are coming in right now.

Step 2 — Detect outliers on the fly. With those statistics, the channels whose magnitude shoots up relative to the tensor median are identified; the typical criterion is a threshold of the form “magnitude > k × median”. These are the channels that, if quantised with the same scale as the rest, blow up the error.

Step 3 — Derive scales and segregate. For normal channels a scale is computed that exploits the range; for the outliers a different treatment is applied, a scale of their own, or keeping them at higher precision, in the style of on-the-fly mixed precision. It is the same philosophy as LLM.int8() (segregating outliers to FP16) or AWQ (scaling salient channels), but with the threshold and the scales recomputed over the current input, not frozen since calibration.

Step 4 — Quantise and multiply. With the fresh scales, the tensor is quantised and the GEMM runs. The activations entering the next layer compensate for the rescaling, just as in AWQ, so that the maths cancels out.

The key difference with AWQ is not in what is done (protecting activation outliers) but in when and against what: AWQ decides once, offline, against a corpus; TTQ decides at every step, on the fly, against real traffic. It is the transfer to inference of the “test-time” idea: adapting the computation to the specific sample in front of you instead of to a precomputed average.

The maths that matter

The error of quantising an outlier with the wrong scale

Let us recall affine uniform quantisation from the base post: an integer code q = round(x/s) - z with scale s and zero-point z, and reconstruction x̂ = s·(q + z). For a b-bit quantiser with a symmetric range, the scale that covers a tensor of maximum magnitude M is approximately s = M / (2^{b-1} - 1). The rounding error of each element is bounded by half a scale: |x - x̂| ≤ s/2.

Here is the outlier problem. The scale s is chosen to cover the largest value in the group. If one channel has 30× the median magnitude and you share a single scale with the rest of the tensor, that magnitude rules: M is the outlier, so s is inflated 30× relative to what the majority would need. The absolute rounding error of the normal values rises proportionally.

A concrete calculation. Take a group where the median magnitude is 1.0 and an outlier channel is 30.0, quantised to INT4 (b = 4, levels ±7):

With a shared scale, s = 30 / 7 ≈ 4.29. The rounding error of a typical value (magnitude ~1) is up to s/2 ≈ 2.14. In other words, the error on the normal values is of the order of their own value: the outlier has destroyed the resolution of everything else. Relative error of a value of magnitude 1: up to ~214 %.
Segregating the outlier (pulling it out to FP16 or giving it its own scale) and quantising the rest with M = 1, s = 1/7 ≈ 0.143. The error of a typical value drops to s/2 ≈ 0.071, ~7 % relative. Thirty times less error on the majority of the group’s weights.

That is the entire reason for the existence of activation-aware quantisation: to detect and treat separately the ~1 % of channels that, if not segregated, hijack the scale. AWQ does it against the corpus; TTQ does it against the real input. And if the channel that is an outlier in production was not an outlier during calibration, AWQ did not protect it: it quantised real traffic with the inflated scale from the case above. That is where TTQ gains precision.

The overhead: the cost of measuring at every step

The price is symmetric. Computing the per-token statistics, per-channel magnitudes, median or percentile, outlier threshold, scales, means reductions over the activation tensor that did not exist in the forward pass with frozen scales. Let us call:

T = forward time per token with fixed scales (static PTQ), in µs.
Δ = extra per-token cost of deriving the statistics and scales on the fly, in µs.

The relative overhead is simply:

$$\text{overhead} = \frac{\Delta}{T}$$

The key point is that Δ is relatively fixed per step (it depends on the number of channels and layers, not on how much “useful” work the model does), while T scales with the size of the model. That is why the ratio behaves very differently depending on the model:

Large model (say, 70B): T is large, every forward moves tens of GB of weights from HBM. If Δ ≈ 8 µs and T ≈ 800 µs, the overhead is 8/800 = 1 %. Negligible against the saving from quantising.
SLM (say, 1B): T is small, the forward per token is short. With the same Δ ≈ 8 µs and T ≈ 60 µs, the overhead is 8/60 ≈ 13 %. No longer negligible: it eats a good part of what you gained by quantising.

This connects directly to the inverted roofline for small models: on an SLM the fixed per-step costs (kernel launches, synchronisations, overheads that do not scale with the model) weigh proportionally more, because there is less useful work to spread them over. TTQ’s Δ is exactly one of those fixed costs. Going per-batch instead of per-token amortises Δ across all the tokens of the batch and lowers the relative overhead, at the cost of coarser scales; it is the first parameter to touch.

The uncomfortable conclusion: TTQ gives away robustness to distribution but spends part of the acceleration budget on measuring, and in the regime where acceleration is scarcest, the SLMs, the ones most often deployed at the edge, is where that spending hurts most. It is not free; it is a change of currency.

A note of methodological scepticism: arXiv:2603.19296 is from March 2026, very recent, and as of the date of this post there are no broad independent reproductions. Whatever speedup and quality figures circulate should be taken with the same caution as any number without published methodology: what hardware, what batch size, what real measured Δ, against what baseline (well-calibrated or badly calibrated PTQ), in what domain? The conceptual argument, robustness to distribution in exchange for per-step overhead, is solid; the concrete multipliers are pending validation.

What TTQ is NOT: marking it off from the rest of the zoo

TTQ is easily confused with neighbouring techniques. The distinction that matters is that TTQ is the how of deriving the scales, not the format nor the moment in training.

Technique	When the scales are fixed	Needs a calibration corpus	Touches training	Is it a format
Static PTQ (GPTQ, AWQ)	Offline, before deployment	Yes	No	No (it uses INT4/INT8)
QAT	During training	No (training data)	Yes (retrains)	No
FP8 end to end	At runtime, but simple per-tensor scales	Minimal / none	No	Yes (E4M3/E5M2)
TTQ	At runtime, activation-aware per token/batch	No	No	No (orthogonal to the format)

The four distinctions, one by one:

Against static PTQ (GPTQ/AWQ). Same goal (protecting outliers), the same possible format (INT4), but PTQ freezes the decisions offline against a corpus and TTQ recomputes them on the fly. TTQ is, in a sense, “AWQ without the calibration phase, paid for at runtime”.
Against QAT. QAT puts quantisation inside the training loop so that the model learns to be robust to it; it costs a retrain. TTQ does not touch training: it operates on an already trained model, at inference. They are attacks at opposite moments of the pipeline.
Against FP8 end to end. FP8 is a format with its own logarithmic range; its “dynamic scaling” computes a simple per-tensor scalar at runtime, but it does not do activation-aware per-channel outlier detection. TTQ could, conceptually, derive scales on the fly for an FP8 or INT4 quantiser: it is orthogonal to the format.
TTQ is orthogonal to the format. It decides how to obtain s, not how many bits you store q in. You can imagine “TTQ over INT4” or “TTQ over FP8”. What defines TTQ is the source of the scale, real activations on the fly, not the width of the code.

When it pays off (and when it does not)

TTQ is not a universal replacement for AWQ. It is a tool for a specific deployment profile. It pays off when:

You have no calibration pipeline. You want to deploy a quantised model now, without assembling the calibration dataset, running the offline pass or validating that the corpus represents the traffic. TTQ cuts out that whole phase: you load the model and serve.
The traffic distribution is shifting or unknown. An assistant that one day receives code and the next legal contracts in another language. No fixed calibration covers both well; on-the-fly adaptation follows the distribution without recalibrating.
Multitenant with no representative corpus. You serve the same model to clients with disparate domains. No single corpus represents all of them; any fixed calibration creates winners and losers among tenants. TTQ adjusts to each input, whichever tenant it comes from.

It does not pay off when:

You have a stable domain and a good calibration corpus. If your traffic is homogeneous and representative, offline AWQ gives you the same quality with zero runtime overhead. Paying Δ on every token to relearn what a corpus already captured is waste.
You serve SLMs with a tight latency SLA. This is exactly the case where Δ/T is high. If the model is small and TPOT matters, the overhead of measuring can wipe out the gain from quantising. Measure your real Δ before assuming it adds up.
The batch is large and compute-bound. With high concurrency the forward is no longer memory-bound and the cost of the extra reductions competes worse; at the very least, amortise Δ per batch.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The natural case for the 4090 is the SLM, Qwen 3 1.5B, Llama 3 8B AWQ-INT4, serving at low concurrency. That is precisely the regime where TTQ is riskiest: T per token is small and the 4090 has no accelerated native FP8 (we discussed this in the quantization post), so TTQ’s extra reductions run on CUDA cores competing for the same time. Here the question is not “does it improve quality?” but “does the overhead leave me an acceptable TPOT?”. If the traffic is homogeneous, offline AWQ wins on simplicity and latency. TTQ only justifies its Δ if the input distribution is genuinely unpredictable and the degradation from fixed calibration is measurable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here the calculation partly flips. With large models T is high and Δ/T drops to the low single-digit percentage range, so TTQ’s overhead is more digestible. The strong use case is multitenant: a cluster serving a large model to clients with heterogeneous domains, where no calibration corpus satisfies everyone. There, TTQ’s robustness to distribution has real value and the overhead is diluted in a big forward. Even so, on an H100 with native FP8, the baseline to beat is demanding: static FP8 barely loses quality (see the table in the quantization post) and costs nothing at runtime. TTQ has to show that its robustness gain on the outlier tenants outweighs what it gives away in overhead. With a paper from March 2026 and no reproductions, that demonstration is pending.

What we have not covered

The memory cost of the on-the-fly statistics: per-channel buffers, their impact on the footprint and on cache pressure.
Interaction with continuous batching: how scales are derived when a batch mixes requests from different domains in the same step.
TTQ + speculative decoding: whether the draft and the target derive scales on the fly separately, and how that affects the acceptance rate.
Numerical stability: what happens when a batch contains a one-off extreme outlier that inflates the scale for every token of that step.

References

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly (March 2026). https://arxiv.org/abs/2603.19296
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024). https://arxiv.org/abs/2306.00978
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023). https://arxiv.org/abs/2210.17323
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023). https://arxiv.org/abs/2211.10438
Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022). https://arxiv.org/abs/2208.07339

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Wed, 27 May 2026 11:30:00 +0200

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.

Awq on lo0 — Tech Blog

Test-time quantization: quantising on the fly with no calibration dataset

TL;DR

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

The problem TTQ solves: fixed calibration ages with the traffic

The TTQ mechanism: measure the real activations and scale on the fly

The maths that matter

The error of quantising an outlier with the wrong scale

The overhead: the cost of measuring at every step

What TTQ is NOT: marking it off from the rest of the zoo

When it pays off (and when it does not)

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

TL;DR

You are here: DEPLOY

The analogy: JPEG with an edge detector

The minimum maths: scale and zero-point

Numerical example

PTQ vs QAT: when quantisation happens

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

Microscaling: NVFP4 and MXFP4

INT4: GPTQ vs AWQ

GPTQ (Frantar et al. 2022)

AWQ (Lin et al. 2023)

Which one to choose

GGUF: the llama.cpp ecosystem

KV cache quantization

Measured quality loss (Llama 3.1 70B Instruct, reference)

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered (upcoming articles)

See also

References