Flashattention on lo0 — Tech Blog

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

Mon, 08 Jun 2026 05:40:00 +0200

Part of the under the engine series. The PagedAttention post explained where the KV lives (in paged blocks). This one explains who reads it and how: the attention kernel. And it connects with FlashAttention v1-v4, which took that kernel apart from the inside; here we look at the level above, how vLLM chooses between several kernels and why it needs more than one.

TL;DR

An LLM forward is, for the most part, standard matrix multiplications that any library does well. The exception that decides performance is attention, and having one good kernel is not enough: you need two, because the two phases of inference are physically opposite problems. Prefill processes the whole prompt: many queries against many keys, dense and compute-bound, the home ground of FlashAttention’s IO-aware tiling. Decode generates one token: a single query against all the accumulated KV, thin and memory-bound, where the only thing that matters is saturating HBM bandwidth while reading the paged KV. That is why vLLM does not have “the attention kernel” but a switchable backend (FLASH_ATTN, FLASHINFER, TRITON_ATTN and others) plus logic that picks according to the GPU: by default FA4 on Blackwell (SM100), FA3 on Hopper (SM90), FA2 elsewhere, with FlashInfer as the alternative that compiles bespoke kernels (JIT) and knows how to do cascade attention for shared prefixes. This post explains why prefill and decode are opposites (with arithmetic intensity), how the backend reads paged KV, how the engine chooses, what FlashInfer brings, the 10 knobs and the trap of pinning a backend blindly. On the generic 4×H100 SXM cluster.

Where you are: the specialist, not the commis

In the kitchen, almost all the work is chopping and sautéing: standard operations any competent commis chef executes, the matrix multiplications of the feed-forward layers and the projections. There is one dish that is not delegated: the signature dish, the one that defines the restaurant. That dish is attention, and it has a quirk: it is cooked in two radically different ways depending on the moment of service.

During prefill, when a new order arrives with its whole prompt, you cook big: a lot of raw material at once, a lot of fire, an intense operation that fills the stoves. During decode, when a table asks for “one more dish”, you cook à la carte: a single dish, but you have to go to the larder and bring back every ingredient that table has accumulated over the whole meal. One is a problem of firepower; the other, of larder speed. The same specialist does not do both well. That is why vLLM has several, and a chef who decides which one steps up depending on the GPU and the phase. That is the attention backend.

Why prefill and decode are opposite problems

This is the central idea, and one sum proves it: arithmetic intensity (FLOPs per byte read). An operation with high intensity is limited by compute; one with low intensity, by memory.

Prefill. We serve $N$ queries (the whole prompt) against $N$ keys. The $QK^\top$ operation and the $\text{softmax}\cdot V$ do on the order of $N^2 d$ FLOPs and read on the order of $N d$ of data. Intensity grows with $N$:

$$I_\text{prefill} \sim \frac{N^2 d}{N d} = N$$

With large $N$ (a prompt of thousands of tokens), intensity is high: compute-bound. This is where FlashAttention’s tiling squeezes the tensor cores and where you get close to the GPU’s peak TFLOPS.

Decode. We serve one query (the new token) against $L$ keys (all the accumulated KV). FLOPs on the order of $L d$; bytes read on the order of $L d s$ (the entire KV has to be read from HBM). Intensity is:

$$I_\text{decode} \sim \frac{L d}{L d s} = \frac{1}{s} \quad (\approx 0.5 \text{ FLOP/byte in FP16})$$

Constant and tiny: memory-bound. The decode kernel is not limited by how much the GPU can compute but by how fast it reads the KV from HBM. It makes no difference that the H100 has 132 idle SMs (see the SM post): the bottleneck is the 3.35 TB/s of bandwidth, and the decode kernel exists so as not to waste a single one of those bytes/s.

The design consequence: a kernel optimised for prefill (dense tiling, maximum tensor-core occupancy) is not the optimum for decode (coalesced reads of the paged KV, minimum latency). Serious servers have different kernels, or one kernel with two paths. In models with MLA (multi-head latent attention), vLLM goes as far as using separate backends for prefill and decode, selectable independently (attention backends, vLLM).

The scheduler’s trick: prefill and decode in the same forward

Here the circle closes with the scheduler post. Because vLLM V1 mixes requests in prefill and in decode in every step, one single forward has to serve both. The backend receives metadata telling it, for each sequence in the batch, how many queries it brings and how much KV it has to read, and applies the right path to each one. That is why the attention backend and the scheduler are coupled: the first has to digest the heterogeneous batch the second builds.

How the backend reads the paged KV

The kernel does not receive a contiguous KV tensor: it receives the block table from the block manager and does a gather over the physical blocks. This imposes a real constraint on the backend: it has to support the paged layout and vLLM’s block_size. Not every kernel in the world does; the ones vLLM integrates (FlashAttention, FlashInfer, Triton) are adapted to read KV in fixed-size blocks scattered across HBM. That is why you cannot plug in any attention kernel from a paper: it has to speak the language of the pigeonhole larder.

The backends and how the engine chooses

vLLM exposes a backend abstraction with several implementations (vLLM deepwiki):

FLASH_ATTN — the FlashAttention family. By default the version is picked by architecture: FA4 on SM100 (Blackwell), FA3 on SM90 (Hopper), FA2 elsewhere, configurable with flash_attn_version.
FLASHINFER — an attention engine with JIT compilation and specialisable kernels; strong on heterogeneous KV and shared prefixes.
TRITON_ATTN — written in Triton, portable and with no dependency on precompiled CUDA binaries (Triton backend deep dive, vLLM, mar-2026).
Specific backends for MLA and for non-NVIDIA hardware.

Selection is automatic unless you force it with VLLM_ATTENTION_BACKEND. The heuristic tries FlashAttention first; on Blackwell (SM100) the fallback order for MLA is TRT-LLM Ragged → FlashInfer → others; on other GPUs only FlashAttention is considered for the main path (attention backends, vLLM). The decision depends on: architecture (SM), dtype (FP16/BF16/FP8), head dimension, and whether the load needs a feature only one backend has (cascade attention, certain soft caps, FP8 in KV).

What FlashInfer brings: JIT and cascade attention

FlashInfer does not compete with FlashAttention on “being a bit faster”; it attacks a different problem: the heterogeneity of the KV in real serving (FlashInfer, arXiv 2501.01005). Two ideas:

JIT compilation. Instead of a monolithic kernel, FlashInfer generates bespoke kernels for the attention variant, the problem shape and the KV layout you have, injecting functors (query/key/logit transformations, masks). It specialises rather than generalises.

Cascade attention. Here is the jewel for serving with shared prefixes. If $R$ requests share a prefix of $P$ tokens (a common system prompt), naive attention would read that prefix $R$ times. Cascade attention computes it once against the shared prefix and then combines with each request’s own suffix:

$$\text{reads: } \underbrace{R \cdot (P + s_i)}_{\text{naive}} \;\longrightarrow\; \underbrace{P + \textstyle\sum_i s_i}_{\text{cascade}}$$

With $R=50$ requests and a prefix $P=1000$, that is reading 50,000 prefix tokens versus 1,000. It is the natural complement to prefix caching: the block manager shares the prefix’s memory, and cascade attention shares the compute of attending over it.

The arithmetic that matters: when switching backend gives you something

The backend only moves the needle where attention is the bottleneck. In memory-bound decode, a kernel that exploits HBM bandwidth better gives a real improvement; in compute-bound prefill with long sequences, FA3/FA4 getting close to the tensor-core peak gives a real improvement. But if your bottleneck is in another layer, launch overhead, a badly sized scheduler, the cold start, then switching backend does not touch that part. The rule, once again: measure the regime before optimising.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	`VLLM_ATTENTION_BACKEND`	force a backend	mismatch with hardware/feature
2	`flash_attn_version` (2/3/4)	FA version	version unsupported on your SM
3	enable FlashInfer	JIT + cascade	initial JIT compilation time
4	cascade attention	prefix compute reuse	only helps with a heavily shared prefix
5	`kv_cache_dtype` (FP8)	FP8 support in the kernel	not every backend/SM supports it
6	`block_size`	layout the kernel must read	consistency with PagedAttention
7	MLA prefill backend	kernel for the dense phase	MLA models only
8	MLA decode backend	kernel for the thin phase	MLA models only
9	soft cap / sliding window	features that restrict backends	fewer kernel options
10	head_dim / variant	which kernels are eligible	exotic models with no support

How it connects to the rest of the stack

With FlashAttention. The FA post explains the kernel from the inside (tiling, online softmax, FA1-4); this is the level above, how vLLM chooses between kernels and why it needs more than one.

With PagedAttention. The backend reads the KV that the block manager places in blocks; it has to speak block-table.

With the scheduler. The scheduler builds mixed prefill+decode batches; the backend has to serve both regimes in a single forward.

With CUDA graphs. Attention kernels are captured in the CUDA graphs; a backend that launches many small kernels benefits more from capture.

With prefix caching. Cascade attention is the compute side of what prefix caching does in memory.

With FP8. Attending over KV in FP8 requires the backend to have the FP8 path; not all of them do on every architecture.

Traps and things that are not what they look like

“FlashInfer is always faster than FlashAttention.” No. FlashInfer wins when its specialisation (cascade, heterogeneous KV, a particular attention variant) applies to your load; in classic dense prefill, FA3/FA4 usually does as well or better. It depends on the regime, there is no universal winner.

“A good attention kernel works for everything.” The underlying error of this post. Prefill and decode are compute-bound and memory-bound respectively; a kernel tuned for one wastes effort on the other. That is why there are separate paths (and separate backends in MLA).

“Decode is compute-bound because the GPU is at 100%.” nvidia-smi at 100% is misleading (see the SM post): decode is memory-bound, the GPU is moving KV, not computing. Optimising decode’s compute is polishing what is not the bottleneck.

“I pin VLLM_ATTENTION_BACKEND and forget about it.” Pinning a backend by hand can leave you on a suboptimal one when you change GPU or version, or force a slow fallback if your hardware does not support what you asked for. Autoselection usually gets it right; pin it only with a measurement that justifies it.

“Cascade attention always helps.” Only with a heavily shared prefix across many concurrent requests. If each request has its own context, there is nothing to share and the overhead of organising the cascade is not amortised.

“The attention backend is the bottleneck, that is why I am slow.” Almost always the bottleneck is higher up (launch, scheduling, memory) or lower down (bandwidth). The backend matters where attention dominates; measure it with nsys/DCGM before changing it.

Conclusion

Of everything an LLM does while generating text, almost all of it is matrix multiplications that any library resolves. Performance is decided in a single kernel, attention, and the surprise is that it is not even one kernel: it is two opposite problems wearing the same name. Prefill wants fire, dense compute over thousands of tokens, and decode wants a fast larder, reading a token’s entire KV with the minimum waste of bandwidth. That is why vLLM did not pick a winning kernel but an abstraction that switches: FlashAttention tuned to each architecture for the general case, FlashInfer compiling bespoke kernels when there is heterogeneity or prefixes to share, Triton for portability. The head chef does not cook the signature dish just one way: he looks at who is ordering and at what point of service, and sends out the right specialist. The lesson for whoever is tuning is the usual one in this series: before changing specialist, make sure the signature dish really is what is holding you back.

References

vLLM, Attention Backends (selección, FA2/3/4 por arquitectura, MLA): https://docs.vllm.ai/en/latest/design/attention_backends/.
vLLM / DeepWiki, FlashAttention and FlashInfer: https://deepwiki.com/vllm-project/vllm/8.2-flashattention-and-flashinfer.
vLLM, Triton Attention Backend Deep Dive (mar-2026): https://vllm.ai/blog/2026-03-04-vllm-triton-backend-deep-dive.
Z. Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (arXiv 2501.01005): https://arxiv.org/abs/2501.01005.
T. Dao, FlashAttention-2 / FlashAttention-3 (kernel IO-aware, async Hopper): https://github.com/Dao-AILab/flash-attention.

FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — IO-awareness, async and Blackwell's asymmetry

Fri, 29 May 2026 17:00:00 +0200

This post complements KV cache: the working memory and PagedAttention deep dive. The KV cache post explains what is stored; PagedAttention, how it is managed in memory; FlashAttention, how the computation is executed. They are three different layers of the same problem and they stack multiplicatively.

You are here: DEPLOY

TL;DR

The standard attention of a transformer has a structural problem on modern GPUs: when you look at it with a profiler, it is not compute-bound, it is memory-bound. The S = QK^T matrix of size N × N, with N the sequence length, does not fit in the fast on-chip SRAM and forces round trips to HBM that dominate the total time. FlashAttention is the family of kernels that avoids materialising that matrix by tiling over Q, K and V, computing the softmax block by block with the online version and keeping everything inside SRAM. Each version raises the utilisation ceiling: FA1 (Dao et al. 2022) abolished the N×N matrix and brought memory complexity down to O(N); FA2 (Dao 2023) parallelised along the sequence dimension and cut non-matmul FLOPs to get close to 70 % utilisation on A100; FA3 (Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao 2024) exploited three Hopper-specific mechanisms, async WGMMA, TMA and FP8, to reach 740 TFLOPS BF16 (75 % of peak) and 1.2 PFLOPS FP8 on H100; FA4 (March 2026) rewrote the kernel from scratch for Blackwell, where the tensor core scales 2.25× but the SFU (where the softmax exp runs) and the SMEM bandwidth do not scale at all. The answer is a software-emulated exponential that runs on tensor cores. The result on B200: 1,605 TFLOPS BF16, 1.3× faster than cuDNN 9.13 and 2× faster than FA3 on the same GPU. This post takes apart the why (memory roofline, IO complexity), the master analogy of the librarian, the minimum mathematics and the real numbers on H100 and B200.

The analogy: the librarian who never clears the desk

A large library with two zones: a very fast but small work desk (the librarian’s desk, 200 books fit on top of it), and a giant shelving system spanning three floors where everything lives (50 million books). The librarian has to cross-reference all the books in one thematic room and produce a summary.

A naive librarian does it the direct way: fetches every relevant book from the shelves, piles them on the work desk, and since they do not fit, leaves half of them on the floor, on the chair, on top of boxes. The day is spent running between the floor and the desk, opening and closing books, never able to concentrate. The work desk is extremely fast but underused, because most of the time the librarian is moving books between the floor and the desk. That is what standard attention does: it materialises the S = QK^T matrix in HBM and keeps coming back for it piece by piece.

A FlashAttention v1 librarian changes strategy: asks for one shelf at a time, brings it to the desk, reads what is needed, jots notes in a compact notebook (“from this shelf I care about this, this and this, with these relative weights”), returns the shelf to its place and brings the next one. The notebook is the only thing that carries over between shelves. Nothing is ever piled higher than the desk can hold. The trick that makes this possible is the online softmax: instead of needing all the content at once in order to normalise, it keeps a running max and a running sum that are updated shelf by shelf.

A FlashAttention v2 librarian realises that several topics can be worked in parallel, because the desk is big and the shelves are independent along some axes. Three assistants are put to work, each with a notebook, each covering a different block of the room, and the results are combined at the end.

A FlashAttention v3 librarian manages to mechanise the flow: a conveyor belt with two stations is installed. While station A reads the current shelf and takes notes, station B already has the next shelf in transit from the shelving system. When A finishes, B hands over the new shelf with no waiting. This is the ping-pong producer/consumer pattern: the TMA offloads the data loading and the consumer warps do the work in parallel with the loads. On top of that, the notes are written in lower-precision shorthand (FP8), because the pages that matter have already had an orthogonal pre-treatment that stops them losing precision where it hurts.

A FlashAttention v4 librarian discovers something new about the remodelled library (Blackwell): two much faster conveyor belts have been installed (tensor cores 2.25×), but the shorthand typewriter has not been upgraded (the SFU is unchanged). Now the bottleneck is writing the notes, not fetching the shelves. The fix is elegant: instead of using the shorthand machine, the notes are written with polynomial formulas that the tensor core itself can evaluate (software-emulated exponential). The belt no longer sits idle waiting for the machine.

The analogy holds up with four mappings:

Work desk = SRAM per SM (228 KB on H100, 256 KB on B200).
Giant shelving system = HBM (3.35 TB/s on H100, 8 TB/s on B200).
Notebook with running max and sum = online softmax stats (m, ℓ).
Conveyor belt with two stations = TMA + WGMMA producer/consumer pipeline.

Why standard attention was the bottleneck

The naive intuition, “attention is matmuls, and GPUs are good at matmuls”, is correct but incomplete. There are two matmuls (QK^T and then softmax(S) V), and in between a non-matmul operation (softmax) that requires materialising the intermediate matrix S of size N × N.

Modern GPUs have a very concrete roofline. For H100 SXM5:

Compute peak: 989 TFLOPS BF16 (tensor core, dense, without sparsity).
Memory bandwidth: 3.35 TB/s HBM3.
Break-even point (arithmetic intensity break-even): 989 × 10¹² / (3.35 × 10¹²) ≈ 295 FLOP/byte.

Any operation that fails to perform 295 operations for every byte it moves from HBM is memory-bound: the tensor core sits waiting for data. On B200 the ratio is similar (~281 FLOP/byte) because both compute and bandwidth went up.

Standard attention materialising S reads the matrix twice (once for the softmax, once to multiply by V) and writes it once. For Llama 3 70B with head dim d = 128 and context N = 128K:

Matrix S per head per layer: N × N × 2 bytes = 128K × 128K × 2 = 34.36 GB.
80 layers × 64 Q-heads → aggregate HBM traffic (if it were materialised serially) on the order of TBs, prohibitive in transient.
Even if it is not all materialised at once, the round trips dominate the time: effective arithmetic intensity well below 295 → memory-bound operation → tensor core underused at around 25 %.

FlashAttention does not change the mathematics of attention, it changes the order of the operations so that S is never materialised in HBM. Its IO complexity is Θ(N²·d²/M) with M = SRAM size per SM, against Θ(N·d + N²) for standard attention. With d = 128 and M = 228 KB: an approximate reduction factor of M/d² ≈ 14× less HBM traffic. That is what moves the operation from memory-bound to nearly compute-bound.

The trick that made it all possible: online softmax

Without online softmax there is no FlashAttention. The idea comes from Milakov and Gimelshein, 2018 (the paper “Online normalizer calculation for softmax”, arXiv:1805.02867), and it allows softmax([x_1, ..., x_N]) to be computed in one incremental pass without needing to know the global maximum before starting.

The standard softmax is:

$$\text{softmax}(x_i) = \frac{e^{x_i - m}}{\sum_{j} e^{x_j - m}}, \quad m = \max_j x_j$$

The online trick: keep a running maximum m^{(t)} and a running sum ℓ^{(t)}. When a new block of values arrives with local maximum m_{\text{new}}:

$$m^{(t+1)} = \max(m^{(t)}, m_{\text{new}})$$ $$\ell^{(t+1)} = e^{m^{(t)} - m^{(t+1)}} \cdot \ell^{(t)} + \sum_{j \in \text{new}} e^{x_j - m^{(t+1)}}$$

And the accumulated partial outputs are rescaled by the same factor e^{m^{(t)} - m^{(t+1)}}. At the end, dividing by the final ℓ gives exactly the same result as the standard softmax. It is mathematically exact, not an approximation.

This is what allows K to be traversed block by block without materialising the whole S matrix. Each block updates the stats and the accumulated outputs, and is then discarded. The desk never fills up.

The four versions (May 2026)

	FA1 (2022)	FA2 (2023)	FA3 (2024)	FA4 (2026)
Target GPU	A100 / Ampere	A100 / H100	H100 / Hopper	B200 / Blackwell
Core idea	Tiling + online softmax	Sequence parallelism + work partitioning	Async WGMMA + TMA + FP8	Polynomial exp + 2-CTA tensor cores
Memory	O(N) (vs O(N²))	same	same	same
Typical peak util	~25 % A100	~70 % A100, ~35 % H100	75 % H100 BF16, 60 % H100 FP8	71 % B200 BF16
Effective TFLOPS	—	225 TFLOPS A100 BF16	740 H100 BF16, 1,200 H100 FP8	1,605 B200 BF16
Speedup vs previous	2-4× standard	2× FA1	1.5-2× FA2 (BF16), 2.6× (FP8)	2× FA3 on B200
Paper	arXiv:2205.14135	arXiv:2307.08691	arXiv:2407.08608	arXiv:2603.05451

FA1 — changing the order of the operations

Three ideas combined: tiling of Q/K/V into blocks that fit in SRAM, online softmax over those blocks, and recomputation in the backward pass. The N×N matrix S is not stored, only the stats (m, ℓ), and in the backward pass S is recomputed block by block from Q, K and the stats. Result: 7.6× speedup on GPT-2 versus PyTorch standard attention, memory O(N).

FA2 — parallelising along the sequence

FA1 parallelised only over batch × heads. With a small batch (1-4) and models with few heads or aggressive GQA, the GPU was left with idle SMs. FA2 also parallelises along the sequence dimension: different SMs process different stretches of Q at the same time. It also rewrites the algorithm to minimise the non-matmul operations (softmax rescaling), because those do not go through tensor cores. And it improves work partitioning between warps (split-Q instead of split-K reduces shared memory traffic). Result: around 2× over FA1 on H100 and A100, 225 TFLOPS on A100 (72 % MFU). On H100 it stays around 30-35 % of BF16 peak because it does not exploit async WGMMA.

FA3 — the Hopper moment

This is where FlashAttention stops being an algorithm and becomes a Hopper-specific artefact. Three pillars:

Async WGMMA: Hopper’s new tensor core instructions let a warpgroup fire a GEMM and have the rest of the warpgroup do something else (the softmax, for instance) while the tensor core keeps working. This is the trick that unlocks matmul/softmax overlap.
TMA (Tensor Memory Accelerator): dedicated hardware for copying tiles between HBM and SRAM. It frees the SM from computing indices and predicating out-of-bounds accesses, which used to consume SM cycles. It is the equivalent of hiring warehouse staff: the librarian no longer has to carry the books personally.
FP8 with block quantisation + incoherent processing: quantising Q and K to FP8 doubles tensor core throughput. The precision loss is mitigated with two tricks: a per-tile scale (64×d) instead of a whole-tensor scale, and a pre-multiplication by a random orthogonal Hadamard-based matrix that “spreads” the outliers before quantising. Documented result: numerical error 2.6× lower than the FP8 baseline.

These three pillars combine with producer/consumer warp specialisation (producer warps do TMA loads; consumer warps do WGMMA + softmax) and ping-pong scheduling with two warpgroups taking turns so that there are never pipeline bubbles. When WG1 does softmax, WG2 does GEMM; then they swap.

Numbers: 740 TFLOPS BF16 on H100 (75 % of the 989 peak), 1.2 PFLOPS FP8 (60 % of the 1978 FP8 dense peak). For sequences ≥ 1K it beats cuDNN. Speedup over FA2: 1.5-2× BF16, 2.6× FP8.

FA4 — Blackwell’s asymmetry

Blackwell scaled everything unevenly:

Tensor core BF16 throughput: 1 PFLOP H100 → 2.25 PFLOPS B200 (2.25×).
SFU count (where the softmax exp runs): unchanged.
Shared memory bandwidth: unchanged.

In other words, if FA3 runs as-is on a B200 with no changes, the matmul goes twice as fast but the softmax stays exactly the same, and that stalls the pipeline. It was only a matter of time before someone fixed the imbalance.

FA4 (March 2026, same team of Dao + Princeton + Together AI + Meta + NVIDIA + Colfax) is a ground-up rewrite with three ideas:

Software-emulated exponential: a polynomial approximation of exp that runs on the tensor core instead of on the SFU. It loses a little precision (carefully bounded and compensated by the rest of the kernel) but keeps the conveyor belt moving.
Conditional softmax rescaling: avoids rescaling accumulators when the running max does not change significantly. A “lazy” style optimisation: it only pays the cost when the cost is needed.
2-CTA tensor core: two CTAs (Cooperative Thread Arrays) cooperate to feed the tensor cores with larger tiles. It gets more out of Blackwell’s new capabilities.

Written in CuTeDSL (NVIDIA’s CUTLASS Python DSL, not CUDA C++ directly). Result on B200 BF16: 1,605 TFLOPS (71 % of the 2250 peak). 1.3× over cuDNN 9.13. 2.7× over Triton. 2× over FA3 run as-is on B200 (which was the previous baseline). It is the first attention kernel to pass 1 PFLOPS.

Note: there is recurring confusion about “FP4 attention”. Blackwell’s NVFP4/MXFP4 extensions apply to weights, not to attention. FA4 can be combined with NVFP4 weights, but the attention computation itself is still BF16 or FP8 depending on the configuration. FP4 quantisation of QK^T does exist in some proprietary kernels (Fireworks AI’s FireAttention V4 combines it) but it is not standard practice.

Implementations and libraries in 2026

Dao-AILab/flash-attention (the canonical repo): supports SM 8.0 (Ampere) with FA2, SM 9.0 (Hopper) with FA3, SM 10.0 (Blackwell datacenter B100/B200/B300) with FA4. The consumer Blackwell version (5090, SM 12.0) has partial support as this post goes out.
FlashInfer (flashinfer-ai/flashinfer, arXiv:2501.01005): an attention engine aimed at serving (not training). Its conceptual contribution is the Block-Sparse Row (BSR), a unified abstraction covering paged KV cache, the radix tree of prefix caching and the tree masks of speculative decoding. Internally it can call FA2/FA3, cuDNN, CUTLASS or trtllm-gen FMHA kernels depending on the case. It JIT-compiles specific variants at runtime. Integrated into vLLM, SGLang and TensorRT-LLM.
vLLM (May 2026): automatic backend selection by GPU. Default FA4 on SM 10.0+, FA3 on SM 9.0, FA2 on the rest. Fallbacks on Blackwell: TRT-LLM Ragged → FlashInfer → TokenSpeed MLA. For FP8 KV cache on B200, FlashInfer is competitive.
SGLang: uses FlashInfer as its attention backend; RadixAttention is the prefix caching layer on top (a radix tree over the KV cache).
TensorRT-LLM: its own fused kernels (trtllm-gen FMHA). XQA is NVIDIA’s own optimisation for GQA in decode.
PyTorch SDPA and FlexAttention: torch.nn.functional.scaled_dot_product_attention selects the backend automatically. FlexAttention (new) lets you define custom masks declaratively and compiles to kernels that can use FA4 as a backend.
xFormers: still alive but residual. The built-in PyTorch SDPA covers most cases.

Cases where FlashAttention does not help

Very short contexts (N < 512): the tiling and kernel launch overhead does not pay off; cuDNN can win.
Non-standard custom masks: FA ships causal, sliding window and ALiBi only. Arbitrary masks need FlexAttention or the JIT variants of FlashInfer.
Non-standard head dim: FA optimises for d = 64, 128, 256. Odd dimensions (d = 96, d = 192) fall into slow paths.
GQA/MQA with extreme ratios: natively supported, but the speedup versus pure MHA depends on the Q-heads : KV-heads ratio.
Cross-attention: supported but less optimised; self-attention is where the gain is largest.
FP8 without block quantisation or incoherent processing: it loses several points in benchmarks. If your serving framework does not implement the two tricks from the FA3 paper, FP8 attention can be a bad idea.

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, SM 8.9)

The 4090 is Ada Lovelace, not Hopper. It runs neither FA3 nor FA4; it runs FA2. That means around 70 % utilisation in BF16 attention (~250 effective TFLOPS against the 4090’s 330 TFLOPS BF16 peak). It is not a disaster, FA2 is already very good compared with standard attention, but the ceiling is clearly below that of an H100. For consumer deploys on a 4090 with Llama 3 8B BF16 or any 14B-32B INT4 AWQ, FA2 is what you will be using, and it is perfectly reasonable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

This is where FA3 shines and it is what vLLM/SGLang/TRT-LLM will select by default. Two common configurations:

Llama 3 70B FP8 with FA3 FP8 attention: 1.2 PFLOPS peak on the GPU, aggregate cluster throughput on the order of 8,000-12,000 tokens/s at medium batch depending on TP and context. For FP8 attention to deliver its full performance it is crucial to use the block quantisation + incoherent processing techniques from the FA3 paper (they are enabled by default in vLLM).
DeepSeek-V3 671B FP8 + MLA with FlashInfer: DeepSeek uses Multi-head Latent Attention (MLA), a different variant of standard attention. FlashInfer has specific kernels (FlashMLA). The typical stack is vLLM/SGLang + FlashInfer + FlashMLA + an FA3 fallback for the non-MLA layers.

If the infrastructure is Blackwell (B200/B300, which some clusters start receiving in 2026), FA4 is the right option and it should be enabled by default in vLLM 0.16+ and SGLang 0.5.11+.

What we have not covered

MLA (Multi-head Latent Attention) from DeepSeek and the specific FlashMLA kernels: they optimise KV cache compression but require different kernels.
Flexible masking and the FlexAttention use cases (PyTorch 2.5+): how to declare arbitrary masks without paying the cost of a custom kernel.
Hardware assistance for sparse attention (NVIDIA 2:4 sparse tensor cores) and why sparse attention has not established itself as a higher ceiling than dense FA.
FA in the backward pass of fine-tuning: this post focuses on inference, but FA3/FA4 also go through the backward pass and they are what makes training models with long contexts viable on H100/B200.

References

Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. https://arxiv.org/abs/2205.14135
Dao, T. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. 2023. https://arxiv.org/abs/2307.08691
Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., Dao, T. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision. NeurIPS 2024. https://arxiv.org/abs/2407.08608
Dao, T. et al. FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling. 2026. https://arxiv.org/abs/2603.05451
Milakov, M., Gimelshein, N. Online normalizer calculation for softmax. 2018. https://arxiv.org/abs/1805.02867
Ye, Z. et al. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving. MLSys 2025. https://arxiv.org/abs/2501.01005
Tri Dao FA3 blog: https://tridao.me/blog/2024/flash3/
Tri Dao FA4 blog: https://tridao.me/blog/2026/flash4/
PyTorch FlashAttention-3 announcement: https://pytorch.org/blog/flashattention-3/
PyTorch FlexAttention + FA4: https://pytorch.org/blog/flexattention-flashattention-4-fast-and-flexible/
Together AI FA4 blog: https://www.together.ai/blog/flashattention-4
Colfax Research FA3: https://research.colfax-intl.com/flashattention-3-fast-and-accurate-attention-with-asynchrony-and-low-precision/
Colfax Research FA4: https://research.colfax-intl.com/flashattention-4-algorithm-and-kernel-pipelining-co-design-for-asymmetric-hardware-scaling/
Repo Dao-AILab/flash-attention: https://github.com/Dao-AILab/flash-attention
Repo flashinfer-ai/flashinfer: https://github.com/flashinfer-ai/flashinfer
vLLM attention backends: https://docs.vllm.ai/en/latest/design/attention_backends/
NVIDIA Hopper Architecture in Depth: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

Flashattention on lo0 — Tech Blog

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

TL;DR

Where you are: the specialist, not the commis

Why prefill and decode are opposite problems

The scheduler’s trick: prefill and decode in the same forward

How the backend reads the paged KV

The backends and how the engine chooses

What FlashInfer brings: JIT and cascade attention

The arithmetic that matters: when switching backend gives you something

The 10 knobs

How it connects to the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — IO-awareness, async and Blackwell's asymmetry

You are here: DEPLOY

TL;DR

The analogy: the librarian who never clears the desk

Why standard attention was the bottleneck

The trick that made it all possible: online softmax

The four versions (May 2026)

FA1 — changing the order of the operations

FA2 — parallelising along the sequence

FA3 — the Hopper moment

FA4 — Blackwell’s asymmetry

Implementations and libraries in 2026

Cases where FlashAttention does not help

Implications on on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, SM 8.9)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References