Fp8 on lo0 — Tech Blog

FP8 end to end: enable it, measure quality and decide with data

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

FP8 is the configuration change with the highest impact per unit of effort available on H100 and Ada Lovelace hardware. On an H100 it enables native FP8 tensor cores: +40-60% decode throughput and ×2 the VRAM available for the KV cache. On an RTX 4090 and an L40 the compute benefit is smaller, but the ×2 VRAM is real and translates directly into twice the concurrency. The risk is quality degradation, which on well-calibrated modern models is <0.5% on standard benchmarks but can be larger on formal reasoning. The correct workflow is not to switch it on and pray: it is to switch it on in staging, run the eval suite, correlate quality with throughput in OTel, and decide with data.

The analogy

A photographer who works with 35 mm negatives and moves to digital. Digital photos take up less space and are processed faster. But a low-resolution photo of a landscape can be indistinguishable from the high-resolution one to the human eye, while a low-resolution photo of text loses letters. Exactly the same trade-off applies to FP8: for tasks where numerical imprecision is averaged out over thousands of activations (conversation, summarisation, RAG), it is practically invisible. For tasks where a single wrong multiplication propagates into an incorrect answer (formal mathematics, critical code), it can be decisive.

The three layers of FP8 in vLLM

FP8 is not a single flag: it is three independent layers that are enabled separately and have different benefits.

Layer 1 — Model weights (--quantization fp8): The model weights are stored and computed in FP8 E4M3. The models have to be pre-quantised (available on HuggingFace with a -FP8 or -fp8 suffix) or quantised at load time with calibration. The benefit: the model takes up half the VRAM and the weight matmuls are 2× faster on an H100.

# Pre-quantised model (recommended for production)
vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8

# Or on-the-fly quantisation (no extra files, somewhat slower on the first tokens)
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --quantization fp8 \
 --kv-cache-dtype auto

Layer 2 — KV cache (--kv-cache-dtype fp8): The K and V tensors of the KV cache are stored in FP8 instead of BF16. It halves the size of the KV cache, doubling the number of tokens that fit in VRAM. It does not affect the model weights.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # dynamic calibration, mandatory to minimise degradation

Layer 3 — Activations (automatic on an H100): On Hopper GPUs, vLLM automatically enables FP8 for the intermediate activations when both previous layers are active. No extra flag is required.

Full production configuration:

vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8 \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --gpu-memory-utilization 0.92 \
 --max-model-len 16384

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

Metric	BF16 baseline	FP8 enabled	Delta
Decode throughput (tok/s, 70B, batch 32)	~1,800	~2,700	+50%
Model VRAM (70B)	140 GB	70 GB	−50%
KV cache VRAM available (on 4×H100)	180 GB	250 GB	+39%
Maximum concurrency (ctx 8K)	~22,500 tok	~31,250 tok	+39%

This is equivalent to an extra replica for free in terms of KV cache capacity.

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

Metric	BF16/Q4 baseline	FP8 KV cache added	Delta
Decode throughput (tok/s, 14B Q4)	~45	~47	+4%
KV cache VRAM available	15 GB	15 GB (same model)	—
Total cache tokens (ctx 8K)	~46,000	~92,000	+100%
Maximum concurrency (ctx 8K)	~5 users	~11 users	+120%

On Ada, the compute benefit is smaller (the FP8 tensor cores do not have the same width as on Hopper), but the ×2 in KV cache capacity is entirely real and translates into twice the possible concurrent users.

The correct workflow: enable, measure, decide

Enabling FP8 directly in production without validating quality is inadequate. The correct workflow has four steps.

Step 1: baseline in staging

Before enabling FP8, record the quality metrics of the current BF16 model. The most reproducible way is to run an eval suite over a fixed dataset and save the results:

# Install lm-evaluation-harness
pip install lm-eval

# BF16 baseline
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/baseline_bf16.json

Step 2: enable FP8 and run the same eval suite

# FP8
lm_eval --model vllm \
 --model_args pretrained=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8,quantization=fp8,kv_cache_dtype=fp8,calculate_kv_scales=true \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/fp8_full.json

Step 3: compute the degradation

# compare_eval.py
import json

with open("results/baseline_bf16.json") as f:
 baseline = json.load(f)
with open("results/fp8_full.json") as f:
 fp8 = json.load(f)

tasks = ["mmlu", "hellaswag", "gsm8k"]
print(f"{'Task':<15} {'BF16':>8} {'FP8':>8} {'Delta':>8} {'OK?':>6}")
print("-" * 50)
for task in tasks:
 b = baseline["results"][task]["acc,none"]
 f = fp8["results"][task]["acc,none"]
 delta = (f - b) / b * 100
 ok = "✓" if abs(delta) < 1.0 else "✗ REVIEW"
 print(f"{task:<15} {b:>8.3f} {f:>8.3f} {delta:>+7.1f}% {ok:>6}")

Decision thresholds documented in MLPerf Inference 2025:

< 0.5% degradation: enable in production without restrictions.
0.5% – 1.5%: enable with active quality monitoring via LLM-as-judge.
> 1.5%: investigate before enabling — a possible calibration problem or an incompatible model.

Step 4: domain eval with LLM-as-judge

Academic benchmarks measure what they measure. Your use case may be different. Adding 200 representative samples from your domain, evaluated by an LLM judge, closes the gap:

# domain_eval.py
from langfuse import Langfuse
from openai import OpenAI

client = Langfuse()
judge = OpenAI(base_url="http://judge-llm:8000/v1", api_key="token")

# Load the 200 curated production samples (prompt + expected answer)
samples = load_domain_samples("eval_dataset_200.json")

scores_bf16, scores_fp8 = [], []
for sample in samples:
 for model_type, endpoint in [("bf16", "http://staging-bf16:8000"), ("fp8", "http://staging-fp8:8000")]:
 response = call_model(endpoint, sample["prompt"])
 score = judge.chat.completions.create(
 model="Qwen/Qwen2.5-72B-Instruct",
 messages=[{
 "role": "user",
 "content": f"Rate this answer from 1 to 5 on accuracy and completeness.\nQuestion: {sample['prompt']}\nExpected answer: {sample['expected']}\nModel answer: {response}\n\nReply with a number from 1 to 5 only."
 }]
 ).choices[0].message.content.strip()

 if model_type == "bf16":
 scores_bf16.append(int(score))
 else:
 scores_fp8.append(int(score))

import numpy as np
print(f"Mean BF16 score: {np.mean(scores_bf16):.2f}")
print(f"Mean FP8 score: {np.mean(scores_fp8):.2f}")
print(f"Degradation: {(np.mean(scores_fp8)-np.mean(scores_bf16))/np.mean(scores_bf16)*100:.1f}%")

OTel + Langfuse correlation: the dashboard that decides

The moment of decision rests on a single dashboard with two signals on the same time axis:

Signal 1 — Throughput (Prometheus):

rate(vllm:generation_tokens_total[5m])

Signal 2 — Mean quality (Langfuse → Prometheus via exporter):

# If you have configured Langfuse with scores exported via OTel
langfuse_score_value{name="llm_judge_domain"}

The expected pattern after enabling FP8: throughput rises by 40-60% and quality stays within ±0.1 points. If quality drops by more than 0.3 points and stays low, there is a real problem.

# Alert: quality drops more than 0.2 points, sustained, after the change
ALERT FP8CalidadDegradada
 IF avg_over_time(langfuse_score_value{name="llm_judge_domain"}[30m])
 < (avg_over_time(langfuse_score_value{name="llm_judge_domain"}[1d] offset 2h) - 0.2)
 FOR 15m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Possible quality degradation after the FP8 configuration change" }

When NOT to enable FP8

FP8 is not always the right answer. The cases where degradation exceeds the acceptable threshold:

Formal mathematical reasoning: GSM8K and MATH are the benchmarks most sensitive to FP8. If your use case is solving mathematical problems or precise financial calculation, measure specifically on these benchmarks before enabling it.

Critical code with tests: numerical precision affects the probability of tokens at key positions in a function. The risk is not that the code “looks” bad, but that it passes superficial tests while having subtle bugs.

Very long contexts without --calculate-kv-scales: without dynamic scale calibration, the accumulated numerical error in the KV cache grows with the context. With --calculate-kv-scales active, the impact is minimal up to 32K tokens.

Small models (<7B): the FP8 conversion overhead can exceed the throughput benefit. The break-even point is around 7B parameters.

References

FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — IO-awareness, async and Blackwell's asymmetry

Fri, 29 May 2026 17:00:00 +0200

This post complements KV cache: the working memory and PagedAttention deep dive. The KV cache post explains what is stored; PagedAttention, how it is managed in memory; FlashAttention, how the computation is executed. They are three different layers of the same problem and they stack multiplicatively.

You are here: DEPLOY

TL;DR

The standard attention of a transformer has a structural problem on modern GPUs: when you look at it with a profiler, it is not compute-bound, it is memory-bound. The S = QK^T matrix of size N × N, with N the sequence length, does not fit in the fast on-chip SRAM and forces round trips to HBM that dominate the total time. FlashAttention is the family of kernels that avoids materialising that matrix by tiling over Q, K and V, computing the softmax block by block with the online version and keeping everything inside SRAM. Each version raises the utilisation ceiling: FA1 (Dao et al. 2022) abolished the N×N matrix and brought memory complexity down to O(N); FA2 (Dao 2023) parallelised along the sequence dimension and cut non-matmul FLOPs to get close to 70 % utilisation on A100; FA3 (Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao 2024) exploited three Hopper-specific mechanisms, async WGMMA, TMA and FP8, to reach 740 TFLOPS BF16 (75 % of peak) and 1.2 PFLOPS FP8 on H100; FA4 (March 2026) rewrote the kernel from scratch for Blackwell, where the tensor core scales 2.25× but the SFU (where the softmax exp runs) and the SMEM bandwidth do not scale at all. The answer is a software-emulated exponential that runs on tensor cores. The result on B200: 1,605 TFLOPS BF16, 1.3× faster than cuDNN 9.13 and 2× faster than FA3 on the same GPU. This post takes apart the why (memory roofline, IO complexity), the master analogy of the librarian, the minimum mathematics and the real numbers on H100 and B200.

The analogy: the librarian who never clears the desk

A large library with two zones: a very fast but small work desk (the librarian’s desk, 200 books fit on top of it), and a giant shelving system spanning three floors where everything lives (50 million books). The librarian has to cross-reference all the books in one thematic room and produce a summary.

A naive librarian does it the direct way: fetches every relevant book from the shelves, piles them on the work desk, and since they do not fit, leaves half of them on the floor, on the chair, on top of boxes. The day is spent running between the floor and the desk, opening and closing books, never able to concentrate. The work desk is extremely fast but underused, because most of the time the librarian is moving books between the floor and the desk. That is what standard attention does: it materialises the S = QK^T matrix in HBM and keeps coming back for it piece by piece.

A FlashAttention v1 librarian changes strategy: asks for one shelf at a time, brings it to the desk, reads what is needed, jots notes in a compact notebook (“from this shelf I care about this, this and this, with these relative weights”), returns the shelf to its place and brings the next one. The notebook is the only thing that carries over between shelves. Nothing is ever piled higher than the desk can hold. The trick that makes this possible is the online softmax: instead of needing all the content at once in order to normalise, it keeps a running max and a running sum that are updated shelf by shelf.

A FlashAttention v2 librarian realises that several topics can be worked in parallel, because the desk is big and the shelves are independent along some axes. Three assistants are put to work, each with a notebook, each covering a different block of the room, and the results are combined at the end.

A FlashAttention v3 librarian manages to mechanise the flow: a conveyor belt with two stations is installed. While station A reads the current shelf and takes notes, station B already has the next shelf in transit from the shelving system. When A finishes, B hands over the new shelf with no waiting. This is the ping-pong producer/consumer pattern: the TMA offloads the data loading and the consumer warps do the work in parallel with the loads. On top of that, the notes are written in lower-precision shorthand (FP8), because the pages that matter have already had an orthogonal pre-treatment that stops them losing precision where it hurts.

A FlashAttention v4 librarian discovers something new about the remodelled library (Blackwell): two much faster conveyor belts have been installed (tensor cores 2.25×), but the shorthand typewriter has not been upgraded (the SFU is unchanged). Now the bottleneck is writing the notes, not fetching the shelves. The fix is elegant: instead of using the shorthand machine, the notes are written with polynomial formulas that the tensor core itself can evaluate (software-emulated exponential). The belt no longer sits idle waiting for the machine.

The analogy holds up with four mappings:

Work desk = SRAM per SM (228 KB on H100, 256 KB on B200).
Giant shelving system = HBM (3.35 TB/s on H100, 8 TB/s on B200).
Notebook with running max and sum = online softmax stats (m, ℓ).
Conveyor belt with two stations = TMA + WGMMA producer/consumer pipeline.

Why standard attention was the bottleneck

The naive intuition, “attention is matmuls, and GPUs are good at matmuls”, is correct but incomplete. There are two matmuls (QK^T and then softmax(S) V), and in between a non-matmul operation (softmax) that requires materialising the intermediate matrix S of size N × N.

Modern GPUs have a very concrete roofline. For H100 SXM5:

Compute peak: 989 TFLOPS BF16 (tensor core, dense, without sparsity).
Memory bandwidth: 3.35 TB/s HBM3.
Break-even point (arithmetic intensity break-even): 989 × 10¹² / (3.35 × 10¹²) ≈ 295 FLOP/byte.

Any operation that fails to perform 295 operations for every byte it moves from HBM is memory-bound: the tensor core sits waiting for data. On B200 the ratio is similar (~281 FLOP/byte) because both compute and bandwidth went up.

Standard attention materialising S reads the matrix twice (once for the softmax, once to multiply by V) and writes it once. For Llama 3 70B with head dim d = 128 and context N = 128K:

Matrix S per head per layer: N × N × 2 bytes = 128K × 128K × 2 = 34.36 GB.
80 layers × 64 Q-heads → aggregate HBM traffic (if it were materialised serially) on the order of TBs, prohibitive in transient.
Even if it is not all materialised at once, the round trips dominate the time: effective arithmetic intensity well below 295 → memory-bound operation → tensor core underused at around 25 %.

FlashAttention does not change the mathematics of attention, it changes the order of the operations so that S is never materialised in HBM. Its IO complexity is Θ(N²·d²/M) with M = SRAM size per SM, against Θ(N·d + N²) for standard attention. With d = 128 and M = 228 KB: an approximate reduction factor of M/d² ≈ 14× less HBM traffic. That is what moves the operation from memory-bound to nearly compute-bound.

The trick that made it all possible: online softmax

Without online softmax there is no FlashAttention. The idea comes from Milakov and Gimelshein, 2018 (the paper “Online normalizer calculation for softmax”, arXiv:1805.02867), and it allows softmax([x_1, ..., x_N]) to be computed in one incremental pass without needing to know the global maximum before starting.

The standard softmax is:

$$\text{softmax}(x_i) = \frac{e^{x_i - m}}{\sum_{j} e^{x_j - m}}, \quad m = \max_j x_j$$

The online trick: keep a running maximum m^{(t)} and a running sum ℓ^{(t)}. When a new block of values arrives with local maximum m_{\text{new}}:

$$m^{(t+1)} = \max(m^{(t)}, m_{\text{new}})$$ $$\ell^{(t+1)} = e^{m^{(t)} - m^{(t+1)}} \cdot \ell^{(t)} + \sum_{j \in \text{new}} e^{x_j - m^{(t+1)}}$$

And the accumulated partial outputs are rescaled by the same factor e^{m^{(t)} - m^{(t+1)}}. At the end, dividing by the final ℓ gives exactly the same result as the standard softmax. It is mathematically exact, not an approximation.

This is what allows K to be traversed block by block without materialising the whole S matrix. Each block updates the stats and the accumulated outputs, and is then discarded. The desk never fills up.

The four versions (May 2026)

	FA1 (2022)	FA2 (2023)	FA3 (2024)	FA4 (2026)
Target GPU	A100 / Ampere	A100 / H100	H100 / Hopper	B200 / Blackwell
Core idea	Tiling + online softmax	Sequence parallelism + work partitioning	Async WGMMA + TMA + FP8	Polynomial exp + 2-CTA tensor cores
Memory	O(N) (vs O(N²))	same	same	same
Typical peak util	~25 % A100	~70 % A100, ~35 % H100	75 % H100 BF16, 60 % H100 FP8	71 % B200 BF16
Effective TFLOPS	—	225 TFLOPS A100 BF16	740 H100 BF16, 1,200 H100 FP8	1,605 B200 BF16
Speedup vs previous	2-4× standard	2× FA1	1.5-2× FA2 (BF16), 2.6× (FP8)	2× FA3 on B200
Paper	arXiv:2205.14135	arXiv:2307.08691	arXiv:2407.08608	arXiv:2603.05451

FA1 — changing the order of the operations

Three ideas combined: tiling of Q/K/V into blocks that fit in SRAM, online softmax over those blocks, and recomputation in the backward pass. The N×N matrix S is not stored, only the stats (m, ℓ), and in the backward pass S is recomputed block by block from Q, K and the stats. Result: 7.6× speedup on GPT-2 versus PyTorch standard attention, memory O(N).

FA2 — parallelising along the sequence

FA1 parallelised only over batch × heads. With a small batch (1-4) and models with few heads or aggressive GQA, the GPU was left with idle SMs. FA2 also parallelises along the sequence dimension: different SMs process different stretches of Q at the same time. It also rewrites the algorithm to minimise the non-matmul operations (softmax rescaling), because those do not go through tensor cores. And it improves work partitioning between warps (split-Q instead of split-K reduces shared memory traffic). Result: around 2× over FA1 on H100 and A100, 225 TFLOPS on A100 (72 % MFU). On H100 it stays around 30-35 % of BF16 peak because it does not exploit async WGMMA.

FA3 — the Hopper moment

This is where FlashAttention stops being an algorithm and becomes a Hopper-specific artefact. Three pillars:

Async WGMMA: Hopper’s new tensor core instructions let a warpgroup fire a GEMM and have the rest of the warpgroup do something else (the softmax, for instance) while the tensor core keeps working. This is the trick that unlocks matmul/softmax overlap.
TMA (Tensor Memory Accelerator): dedicated hardware for copying tiles between HBM and SRAM. It frees the SM from computing indices and predicating out-of-bounds accesses, which used to consume SM cycles. It is the equivalent of hiring warehouse staff: the librarian no longer has to carry the books personally.
FP8 with block quantisation + incoherent processing: quantising Q and K to FP8 doubles tensor core throughput. The precision loss is mitigated with two tricks: a per-tile scale (64×d) instead of a whole-tensor scale, and a pre-multiplication by a random orthogonal Hadamard-based matrix that “spreads” the outliers before quantising. Documented result: numerical error 2.6× lower than the FP8 baseline.

These three pillars combine with producer/consumer warp specialisation (producer warps do TMA loads; consumer warps do WGMMA + softmax) and ping-pong scheduling with two warpgroups taking turns so that there are never pipeline bubbles. When WG1 does softmax, WG2 does GEMM; then they swap.

Numbers: 740 TFLOPS BF16 on H100 (75 % of the 989 peak), 1.2 PFLOPS FP8 (60 % of the 1978 FP8 dense peak). For sequences ≥ 1K it beats cuDNN. Speedup over FA2: 1.5-2× BF16, 2.6× FP8.

FA4 — Blackwell’s asymmetry

Blackwell scaled everything unevenly:

Tensor core BF16 throughput: 1 PFLOP H100 → 2.25 PFLOPS B200 (2.25×).
SFU count (where the softmax exp runs): unchanged.
Shared memory bandwidth: unchanged.

In other words, if FA3 runs as-is on a B200 with no changes, the matmul goes twice as fast but the softmax stays exactly the same, and that stalls the pipeline. It was only a matter of time before someone fixed the imbalance.

FA4 (March 2026, same team of Dao + Princeton + Together AI + Meta + NVIDIA + Colfax) is a ground-up rewrite with three ideas:

Software-emulated exponential: a polynomial approximation of exp that runs on the tensor core instead of on the SFU. It loses a little precision (carefully bounded and compensated by the rest of the kernel) but keeps the conveyor belt moving.
Conditional softmax rescaling: avoids rescaling accumulators when the running max does not change significantly. A “lazy” style optimisation: it only pays the cost when the cost is needed.
2-CTA tensor core: two CTAs (Cooperative Thread Arrays) cooperate to feed the tensor cores with larger tiles. It gets more out of Blackwell’s new capabilities.

Written in CuTeDSL (NVIDIA’s CUTLASS Python DSL, not CUDA C++ directly). Result on B200 BF16: 1,605 TFLOPS (71 % of the 2250 peak). 1.3× over cuDNN 9.13. 2.7× over Triton. 2× over FA3 run as-is on B200 (which was the previous baseline). It is the first attention kernel to pass 1 PFLOPS.

Note: there is recurring confusion about “FP4 attention”. Blackwell’s NVFP4/MXFP4 extensions apply to weights, not to attention. FA4 can be combined with NVFP4 weights, but the attention computation itself is still BF16 or FP8 depending on the configuration. FP4 quantisation of QK^T does exist in some proprietary kernels (Fireworks AI’s FireAttention V4 combines it) but it is not standard practice.

Implementations and libraries in 2026

Dao-AILab/flash-attention (the canonical repo): supports SM 8.0 (Ampere) with FA2, SM 9.0 (Hopper) with FA3, SM 10.0 (Blackwell datacenter B100/B200/B300) with FA4. The consumer Blackwell version (5090, SM 12.0) has partial support as this post goes out.
FlashInfer (flashinfer-ai/flashinfer, arXiv:2501.01005): an attention engine aimed at serving (not training). Its conceptual contribution is the Block-Sparse Row (BSR), a unified abstraction covering paged KV cache, the radix tree of prefix caching and the tree masks of speculative decoding. Internally it can call FA2/FA3, cuDNN, CUTLASS or trtllm-gen FMHA kernels depending on the case. It JIT-compiles specific variants at runtime. Integrated into vLLM, SGLang and TensorRT-LLM.
vLLM (May 2026): automatic backend selection by GPU. Default FA4 on SM 10.0+, FA3 on SM 9.0, FA2 on the rest. Fallbacks on Blackwell: TRT-LLM Ragged → FlashInfer → TokenSpeed MLA. For FP8 KV cache on B200, FlashInfer is competitive.
SGLang: uses FlashInfer as its attention backend; RadixAttention is the prefix caching layer on top (a radix tree over the KV cache).
TensorRT-LLM: its own fused kernels (trtllm-gen FMHA). XQA is NVIDIA’s own optimisation for GQA in decode.
PyTorch SDPA and FlexAttention: torch.nn.functional.scaled_dot_product_attention selects the backend automatically. FlexAttention (new) lets you define custom masks declaratively and compiles to kernels that can use FA4 as a backend.
xFormers: still alive but residual. The built-in PyTorch SDPA covers most cases.

Cases where FlashAttention does not help

Very short contexts (N < 512): the tiling and kernel launch overhead does not pay off; cuDNN can win.
Non-standard custom masks: FA ships causal, sliding window and ALiBi only. Arbitrary masks need FlexAttention or the JIT variants of FlashInfer.
Non-standard head dim: FA optimises for d = 64, 128, 256. Odd dimensions (d = 96, d = 192) fall into slow paths.
GQA/MQA with extreme ratios: natively supported, but the speedup versus pure MHA depends on the Q-heads : KV-heads ratio.
Cross-attention: supported but less optimised; self-attention is where the gain is largest.
FP8 without block quantisation or incoherent processing: it loses several points in benchmarks. If your serving framework does not implement the two tricks from the FA3 paper, FP8 attention can be a bad idea.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, SM 8.9)

The 4090 is Ada Lovelace, not Hopper. It runs neither FA3 nor FA4; it runs FA2. That means around 70 % utilisation in BF16 attention (~250 effective TFLOPS against the 4090’s 330 TFLOPS BF16 peak). It is not a disaster, FA2 is already very good compared with standard attention, but the ceiling is clearly below that of an H100. For consumer deploys on a 4090 with Llama 3 8B BF16 or any 14B-32B INT4 AWQ, FA2 is what you will be using, and it is perfectly reasonable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

This is where FA3 shines and it is what vLLM/SGLang/TRT-LLM will select by default. Two common configurations:

Llama 3 70B FP8 with FA3 FP8 attention: 1.2 PFLOPS peak on the GPU, aggregate cluster throughput on the order of 8,000-12,000 tokens/s at medium batch depending on TP and context. For FP8 attention to deliver its full performance it is crucial to use the block quantisation + incoherent processing techniques from the FA3 paper (they are enabled by default in vLLM).
DeepSeek-V3 671B FP8 + MLA with FlashInfer: DeepSeek uses Multi-head Latent Attention (MLA), a different variant of standard attention. FlashInfer has specific kernels (FlashMLA). The typical stack is vLLM/SGLang + FlashInfer + FlashMLA + an FA3 fallback for the non-MLA layers.

If the infrastructure is Blackwell (B200/B300, which some clusters start receiving in 2026), FA4 is the right option and it should be enabled by default in vLLM 0.16+ and SGLang 0.5.11+.

What we have not covered

MLA (Multi-head Latent Attention) from DeepSeek and the specific FlashMLA kernels: they optimise KV cache compression but require different kernels.
Flexible masking and the FlexAttention use cases (PyTorch 2.5+): how to declare arbitrary masks without paying the cost of a custom kernel.
Hardware assistance for sparse attention (NVIDIA 2:4 sparse tensor cores) and why sparse attention has not established itself as a higher ceiling than dense FA.
FA in the backward pass of fine-tuning: this post focuses on inference, but FA3/FA4 also go through the backward pass and they are what makes training models with long contexts viable on H100/B200.

References

Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. https://arxiv.org/abs/2205.14135
Dao, T. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. 2023. https://arxiv.org/abs/2307.08691
Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. NeurIPS 2024. https://arxiv.org/abs/2407.08608
Dao, T. et al. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. 2026. https://arxiv.org/abs/2603.05451
Milakov, M., Gimelshein, N. Online normalizer calculation for softmax. 2018. https://arxiv.org/abs/1805.02867
Ye, Z. et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. MLSys 2025. https://arxiv.org/abs/2501.01005
Tri Dao FA3 blog: https://tridao.me/blog/2024/flash3/
Tri Dao FA4 blog: https://tridao.me/blog/2026/flash4/
PyTorch FlashAttention-3 announcement: https://pytorch.org/blog/flashattention-3/
PyTorch FlexAttention + FA4: https://pytorch.org/blog/flexattention-flashattention-4-fast-and-flexible/
Together AI FA4 blog: https://www.together.ai/blog/flashattention-4
Colfax Research FA3: https://research.colfax-intl.com/flashattention-3-fast-and-accurate-attention-with-asynchrony-and-low-precision/
Colfax Research FA4: https://research.colfax-intl.com/flashattention-4-algorithm-and-kernel-pipelining-co-design-for-asymmetric-hardware-scaling/
Repo Dao-AILab/flash-attention: https://github.com/Dao-AILab/flash-attention
Repo flashinfer-ai/flashinfer: https://github.com/flashinfer-ai/flashinfer
vLLM attention backends: https://docs.vllm.ai/en/latest/design/attention_backends/
NVIDIA Hopper Architecture in Depth: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Wed, 27 May 2026 11:30:00 +0200

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.

Fp8 on lo0 — Tech Blog

FP8 end to end: enable it, measure quality and decide with data

TL;DR

The analogy

The three layers of FP8 in vLLM

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

The correct workflow: enable, measure, decide

Step 1: baseline in staging

Step 2: enable FP8 and run the same eval suite

Step 3: compute the degradation

Step 4: domain eval with LLM-as-judge

OTel + Langfuse correlation: the dashboard that decides

When NOT to enable FP8

See also

In this same series

References

FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — IO-awareness, async and Blackwell's asymmetry

You are here: DEPLOY

TL;DR

The analogy: the librarian who never clears the desk

Why standard attention was the bottleneck

The trick that made it all possible: online softmax

The four versions (May 2026)

FA1 — changing the order of the operations

FA2 — parallelising along the sequence

FA3 — the Hopper moment

FA4 — Blackwell’s asymmetry

Implementations and libraries in 2026

Cases where FlashAttention does not help

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, SM 8.9)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

TL;DR

You are here: DEPLOY

The analogy: JPEG with an edge detector

The minimum maths: scale and zero-point

Numerical example

PTQ vs QAT: when quantisation happens

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

Microscaling: NVFP4 and MXFP4

INT4: GPTQ vs AWQ

GPTQ (Frantar et al. 2022)

AWQ (Lin et al. 2023)

Which one to choose

GGUF: the llama.cpp ecosystem

KV cache quantization

Measured quality loss (Llama 3.1 70B Instruct, reference)

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered (upcoming articles)

See also

References