Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Contents

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.