Quantization on lo0 — Tech Blog

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

Tue, 16 Jun 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

Seven independent levers act on the numerator or the denominator of the identity $\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}$. Applied in order of cost-benefit: quantisation (FP16→FP8) cuts J/token by 30–50% without changing hardware; power capping to 70% of TDP reduces power by 30% with less than a 10% drop in throughput on memory-bound loads; DVFS in the decode phase saves a further 22–45%; continuous batching lowers J/token by 25–40% against unoptimised stacks; speculative decoding trims up to 29% off J/token at low batch; prefix caching eliminates the prefill recompute for repeated prompts; and scheduling to clean hours shifts the carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100 SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect can reach a 73% reduction against the unoptimised baseline (arXiv 2504.17674).

The reference identity

$$\text{J/token} = \frac{\overline{P}\ [\text{W}]}{\text{throughput}\ [\text{tok/s}]}$$

where $\overline{P}$ is the average power over the measurement window (see the methodology in energy per token). The levers act on one of the two factors:

They reduce $\overline{P}$: power capping, DVFS, scheduling to low-intensity hours.
They raise throughput (the denominator): quantisation, continuous batching, speculative decoding, prefix caching, chunked prefill, model selection.
Both at once: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher throughput at similar or lower power).

The expanded formula including infrastructure overhead:

$$\text{J/token}_{\text{effective}} = \frac{\overline{P}_{\text{GPU}} + \overline{P}_{\text{rest of node}}}{\text{throughput}_{\text{tok/s}}} \times \text{PUE}$$

For the lever calculations the work is done on $\overline{P}_{\text{GPU}}$ for clarity; the PUE factor (see energy per token) amplifies or attenuates the effect of each lever proportionally.

Master table: levers × impact × cost × tool

Lever	J/token reduction (%)	Effect on latency	Effect on quality	Main OSS tool	Source
Quantisation FP16→FP8	30–50%	−8–33% TPOT (improvement)	<1% degradation on benchmarks	vLLM, TRT-LLM, llm-compressor	arXiv 2504.17674, baseten.co
Quantisation FP16→INT4 (AWQ/GPTQ)	45–60%	latency improves (lower BW)	1–3% degradation depending on model	AutoAWQ, llm-compressor, vLLM	arXiv 2504.17674
Continuous vs static batching	25–40% vs PyTorch	+5× throughput (improvement)	no impact	vLLM, TGI, TRT-LLM	arXiv 2504.17674, TokenPowerBench
Power capping 70% TDP	20–30% power	<10% throughput (decode)	no impact	nvidia-smi, DCGM, Zeus	arXiv 2604.11391, NERSC docs
DVFS decode-phase	22–45% in decode	1–6% total latency	no impact	nvidia-smi, GreenLLM, DVFS-GPT	arXiv 2501.08219, HAL 05190051
Speculative decoding	up to 29% (batch≤16)	TTFT improves	no demonstrable impact	vLLM, TRT-LLM	arXiv 2602.09113
Prefix caching	50–90% on reusable prefill	TTFT −50–80% (repeated prompts)	no impact	vLLM, SGLang	vLLM docs
Chunked prefill	~10–15% on mixed loads	more stable TTFT	no impact	vLLM (SARATHI)	arXiv 2308.16369
SLM vs dense LLM selection	60–80% (7B vs 70B model)	low latency	degradation varies by task	Ollama, vLLM, llama.cpp	arXiv 2504.17674
Scheduling to clean hours	0% J/token, up to −70% CO₂	no impact on the load	no impact	Kueue, Volcano, cron	ACM SLIT, arXiv 2507.09942

Note: the percentages are against the effective J/token of the load, not against idle board power. The effects are always measured under conditions of equal output quality (same model, same serving stack, same prompt distribution).

Lever 1: quantisation

Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations), which shortens the time spent reading weights from HBM per generated token. Decode is memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width frees bandwidth for higher throughput or allows a larger batch in VRAM.

Format	Bits/weight	BW reduction vs FP32	J/token vs FP16	Tool
FP32	32	1× (reference)	+100% (worse)	—
BF16/FP16	16	2×	reference	vLLM native
FP8 (W8A8)	8	~4×	−30–50%	vLLM, TRT-LLM, llm-compressor
INT4 AWQ (W4A16)	4	~8×	−45–60%	AutoAWQ, llm-compressor
INT4 GPTQ (W4A16)	4	~8×	−40–55%	AutoGPTQ, vLLM

Concrete reference figures:

Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten, baseten.co/blog/33-faster-llm-inference-with-fp8-quantization).
The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token (arXiv 2504.17674).
Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined hardware+quantisation reduction: ~90% (MLCommons, arXiv 2504.17674).

Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU, HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task; INT4 GPTQ without fine calibration, up to 2–5%. Validate with LLM energy leaderboards and your own quality benchmarks.

OSS tooling

# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --quantization fp8 \
 --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
 --model meta-llama/Llama-3-70B-Instruct \
 --recipe w4a16_awq.yaml \
 --output-dir llama-70b-w4a16/

Lever 2: continuous batching

Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays empty until the end of the batch. Continuous batching inserts new requests as soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the throughput grows. The result is a larger denominator at similar power:

$$\text{J/token}_{\text{continuous batch}} = \frac{\overline{P}}{\text{throughput}_{\uparrow}}$$

Measured figures (arXiv 2504.17674, TokenPowerBench arXiv 2512.03024):

vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high batch loads.
Online continuous batching vs static offline: a further −5% of J/token (small but consistent).
Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that, J/token flattens or rises slightly.

OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM: --max-num-seqs (maximum concurrent sequences); tuning it to the available VRAM maximises the effective batch.

Lever 3: power capping

Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the clock (internal DVFS). The efficiency trick: LLM decode is memory-bandwidth-bound; the GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading throughput proportionally.

The power cap vs throughput curve (H100 SXM)

Figures from (arXiv 2604.11391) and NERSC (docs.nersc.gov/jobs/power-capping):

Power cap (W)	% of TDP (700 W)	Relative throughput (memory-bound)	Efficiency (tok/W)
700 W	100%	100%	1.00×
560 W	80%	~95%	~1.19×
490 W	70%	~90–95%	~1.27× (sweet spot)
420 W	60%	~80–85%	~1.33× (latency rises)
350 W	50%	~75% (the H100 reaches maximum BW at 350 W memory-bound)	~1.50×
200 W	29%	~50%	frequency saturation

Key finding from arXiv 2604.11391: the H100 reaches its maximum effective memory bandwidth with only ~350 W on memory-bound loads (STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only ~10% extra throughput for each additional 100 W.

J/token formula with a power cap

$$\text{J/token}(P_{\text{cap}}) = \frac{P_{\text{cap}} \times \eta(P_{\text{cap}})}{\text{throughput}_{\text{base}} \times f(P_{\text{cap}})}$$

where $\eta(P_{\text{cap}})$ is the fraction of power actually consumed under the cap (on memory-bound loads, the GPU no longer consumes the whole cap), and $f(P_{\text{cap}})$ is the resulting fraction of throughput. In the H100’s 490–560 W zone, $\eta \approx 0{.}9$ and $f \approx 0{.}92\text{–}0{.}95$: the numerator falls more than the denominator, so J/token improves.

Commands

# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active

# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
 sudo nvidia-smi -i $i -pl 490
done

With DCGM in production, the cap can be set via dcgmi policy --set powercap:490 or through the NVIDIA GPU Operator (the powerLimit field in the DevicePlugin). Zeus (ml.energy/zeus) can sweep caps automatically and measure the resulting J/token per configuration.

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly (nvidia-smi -lgc to lock the GPU clock) or left for the driver to adjust dynamically according to available power and load. In LLM inference, the asymmetry between phases is the key fact:

Prefill: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
Decode: memory-bandwidth-bound. The SM clock barely matters; what rules is HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

System	Energy saving	Latency penalty	Hardware	Source
DVFS-GPT (HAL)	up to 32% at iso-latency	~1–2%	A100, RTX 4090	HAL 05190051
EcoInfer	up to 25.4% (average 21.5%)	<5%	A100	arXiv MDPI Electronics
DualScale (disaggregated prefill/decode)	22–45% with SLO	1–6%	H100	arXiv 2602.18755
GreenLLM (SLO-aware DVFS)	18–30%	within the SLO	—	arXiv 2508.16449

The paper arXiv 2501.08219 measures that reducing the GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6% latency increase in the decode phase, because decode is memory-bound and the SM frequency is not the bottleneck.

Commands (manual DVFS)

# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without overrunning the latency budget.

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

J/token scales with model size, but sublinearly. For Llama-3 (arXiv 2504.17674):

Model	Parameters	Relative J/token	Factor vs 1B
Llama-3 1B	1B	1.0×	reference
Llama-3 8B	8B	~2.3×	×2.3 (not ×8)
Llama-3 70B	70B	~7.3×	×7.3 (not ×70)
Llama-3 405B	405B	~40×	×40 (not ×405)

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware efficiency (caches, tensor parallelism, kernels) cushions the scaling. Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for the task.

MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which reduces active compute. But there is an implementation trap:

OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes 54.24% more energy per token than the dense model of the same active size (arXiv 2504.17674), because the fused expert kernels are ~19–63% slower than an equivalent dense GEMM.
OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is equal total parameters, fewer active (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison is equal active parameters per token.

Model selection table by use case

Use case	Recommended model	Efficiency rationale
Simple chat, entity extraction	SLM 7–8B FP8	−68% J/token vs 70B
RAG with long contexts	70B FP8 or MoE 8×7B	optimal quality/J at K>2K tokens
Complex code, reasoning	70B FP8 or 405B INT4	quality is non-negotiable
Classification, embeddings	dedicated model (e.g. E5, BGE)	orders of magnitude less J
Deferrable offline batch	large MoE (DeepSeek, Mixtral)	high quality, batch throughput

Lever 6: speculative decoding

Mechanics

Speculative decoding uses a small draft model to propose several tokens, which the target model verifies in parallel in a single forward pass. If the tokens are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

The energy efficiency condition

The J/token benefit depends on batch size (arXiv 2602.09113):

Batch size	Effect on J/token	Reason
1–16 (low batch)	−up to 29%	the draft model fills the waiting time; verification is cheap
32–64 (medium batch)	~0% (neutral)	verification overhead cancels out the gain
128+ (high batch)	+25% (worse)	GPU saturated; the draft model adds overhead without saving

Conclusion: speculative decoding is a J/token lever only in a low-latency regime (batch≤16); in high-throughput serving it is counterproductive in J/token terms (although it reduces TPOT latency).

OSS tooling

vLLM supports speculative decoding with --speculative-model and --num-speculative-tokens. TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as draft for Llama-3 70B).

Lever 7: prefix caching and chunked prefill

Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document context) is identical across requests. Storing and reusing it eliminates the prefill recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt and 200-token user prompts, the saving is ~1,000/(1,000+200) = ~83% of prefill energy. In total J/token, the effect depends on the length of the shared prompt against the generation length.

Enabling it in vLLM:

vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-prefix-caching \
 --max-num-seqs 256

Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with decode tokens from other requests. The benefit: the prefill of a long prompt no longer blocks the decode of other requests (reducing TPOT spikes), which allows higher GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node’s peak power (fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less J/token on mixed loads with long prompts (arXiv 2308.16369).

# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-chunked-prefill \
 --max-num-batched-tokens 512

Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the gCO₂/token of deferrable load. The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting (ACM SLIT, arXiv 2507.09942):

CO₂ reduction from temporal shifting: up to 34.7% for flexible loads.
With carbon-aware reinforcement learning (Eco-Orchestrator): up to a 70% offset in integrated deployments.
In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the same day (see from watt to carbon).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a carbon-deadline annotation (the maximum tolerable wait) and the scheduler places them in the minimum-intensity hours within that window.

# Kueue annotation for carbon-aware scheduling
metadata:
 annotations:
 kueue.x-k8s.io/carbon-aware: "true"
 kueue.x-k8s.io/carbon-deadline: "8h"

The power cap vs throughput curve: the sweet spot

Reading the chart: the power curve (solid line) falls faster than the throughput curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W gives only ~5% extra throughput; below 420 W, throughput starts falling faster than power. The efficiency sweet spot (minimum J/token) sits at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill saturates SM compute and needs more power to sustain throughput.

How to measure the effect and validate quality

Measurement stack

The full stack for measuring the effect of each lever:

What to measure	Tool	Metric
Real-time GPU power	DCGM (`DCGM_FI_DEV_POWER_USAGE`)	W per GPU, 100ms
J/token in production	Kepler + vLLM tokens	`rate(kepler_joules) / rate(vllm_tokens)`
J/token on the bench	Zeus (`ZeusMonitor`)	J measured during the benchmark
Throughput	vLLM metrics (`vllm:generation_tokens_total`)	tok/s
TPOT latency	vLLM (`vllm:e2e_request_latency`)	ms/tok
Output quality	lm-evaluation-harness	MMLU, HumanEval, etc.

See the full stack in energy tooling.

Measurement protocol per lever

# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
 /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)

With Zeus, the configuration sweep is automatic:

from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
 # run the benchmark with vLLM FP8
 run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")

Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness):

# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/

The MMLU score difference between FP16 and FP8 with calibration is typically <1%. Without calibration it can be 2–4%.

Pareto decision table (energy / latency / quality)

Lever	J/token	TPOT (latency)	Quality	Implementation cost	Reversibility
FP8 quantisation	↓↓↓	↓ (improves)	~ (−<1%)	Low (1 flag)	High
INT4 AWQ quantisation	↓↓↓↓	↓ (improves)	↓ (−1–3%)	Medium (recalibrate)	High
Continuous batching	↓↓↓	~ (neutral on TPOT)	~	Low (new engine)	High
Power cap 70% TDP	↓↓	↑ <10%	~	Very low (1 command)	Immediate
DVFS decode	↓↓↓	↑ <6%	~	Medium (script + tuning)	High
Speculative decoding	↓ (low batch)	↓ (improves TTFT)	~	Medium (draft model)	High
Prefix caching	↓↓ (prefill)	↓↓ (TTFT improves)	~	Low (1 flag)	High
Chunked prefill	↓	↑ stable TPOT	~	Low (1 flag)	High
SLM (7B vs 70B)	↓↓↓↓	↓↓ (improves a lot)	↓↓ (−variable)	High (quality re-eval)	Medium
MoE (vs dense, same active size)	↑ (+54%)	↑	~	—	—
Scheduling to clean hours	~ J/token, ↓↓ CO₂	~ (for deferrable)	~	Medium (scheduler)	High

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement · ~ neutral · ↑ worsens · ↑↑ worsens significantly.

Stacking levers: the combined effect

The levers are mostly orthogonal and they stack. The maximum combined impact documented in (arXiv 2504.17674):

“Correct application of the relevant inference optimisations can reduce total energy consumption by up to 73% against unoptimised baselines.”

Recommended sequence by cost-benefit:

Optimised engine (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
FP8 quantisation (if the hardware supports it: H100, A100 with limited support): a further −30–50% on the remaining J/token.
Power cap at 70% TDP: −20–30% power with <10% of throughput. Low cost.
Prefix caching: turn it on if there are system prompts or repeated contexts. No cost.
Chunked prefill: if the load has long variable prompts.
DVFS in decode: a further −22–45% if the SLO allows it.
Speculative decoding: only if the batch is low (<16) and TTFT latency matters.
Model selection: move to an SLM if the quality is sufficient (evaluate with lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

Configuration	Power (W)	Throughput (tok/s)	J/token
PyTorch FP16, static batching	~2,100	~400	5.25
vLLM FP16, continuous batching	~2,200	~1,800	1.22
vLLM FP8, continuous batching	~2,100	~2,400	0.875
+ power cap 490 W/GPU (1,960 W node)	~1,960	~2,160	0.907
+ prefix caching (40% hit rate)	~1,960	~2,600	0.754
+ DVFS decode 1,200 MHz	~1,500	~2,400	0.625

The values are illustrative estimates based on the cited papers applied to a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

Reference hardware: TDP and power cap ranges

GPU	TDP (W)	Supported cap range	Decode sweet spot
H100 SXM 80GB	700	200–700 W	~490–560 W
H100 PCIe 80GB	350	100–350 W	~245–280 W
A100 SXM 80GB	400	100–400 W	~280–320 W
A100 PCIe 80GB	300	100–300 W	~210–240 W
L40S 48GB	350	100–350 W	~245–280 W
RTX 5090 (reference, PCIe)	575	~100–575 W	~400–460 W

For the exact ranges: nvidia-smi -q -d POWER | grep -E "Min|Max|Current".

Executive summary by role

Role	First action	Second action	Key tool
Cluster operator	`nvidia-smi -pl [70% TDP]` on every node	Enable persistent mode	nvidia-smi, DCGM
Serving engineer	Migrate to vLLM FP8 with continuous batching	Enable prefix caching	vLLM, llm-compressor
Platform architect	Select the smallest model that meets the quality SLO	Design a carbon-based scheduling policy	lm-evaluation-harness, Kueue
MLOps team	Set up a power cap sweep with Zeus	Instrument J/token in Prometheus	Zeus, DCGM exporter

See the full measurement stack in energy tooling and the J/token methodology in energy per token. For the context of reference energy leaderboards and benchmarks, see LLM energy leaderboards. The connection with full TCO is in GPU utilisation as FinOps.

Sources

arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
Zeus Project (ml.energy) — https://ml.energy/zeus/
Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139

Aggressive quantisation (state of the art): from 4-bit to ternary

Tue, 09 Jun 2026 02:10:00 +0000

This post is the direct continuation of Quantization for LLM inference, which covers the “solved” regime (FP8, INT4 with GPTQ/AWQ). Read it first: here I assume the scale+zero-point maths, what GPTQ and AWQ do, and the PTQ/QAT distinction. What we add is the sub-4-bit frontier, where scalar post-hoc quantisation stops working and you have to change tools.

TL;DR

There is a sharp dividing line around 4 bits. Above it, quantising is a solved problem: INT8 is indistinguishable from BF16, and INT4 with a good method (AWQ, GPTQ) loses 1-2 MMLU points and little else. The method is still the usual one, take each weight, scale it, round it to a short integer, and it works. Below 4 bits, that method collapses: at 2 bits, naive scalar quantisation can double perplexity. The reason is geometric, each weight has only 4 possible values and the rounding error stops being negligible, and the way out is not “round better”, it is to change the representation. State-of-the-art 2-bit methods (AQLM, QuIP#, QTIP) stop quantising individual weights and quantise vectors of weights against dictionaries (codebooks), and they “whiten” the weight matrix to spread its energy and flatten outliers (incoherence processing). Ternary is something else again: BitNet b1.58, with weights in {-1, 0, +1} (~1.58 bits), is not PTQ, it is a model trained natively under that constraint, and it changes the arithmetic of the matmul from multiplications to additions and subtractions, hitting the compute ceiling and the memory ceiling at once. The mental rule: at ≥4-bit you compress the photo; below 4-bit you have to repaint it.

The analogy: the JPEG that cannot be compressed any further

In the quantization post we used the JPEG with an edge detector to explain INT4. The analogy still holds here, but it has to be pushed to its limit.

A JPEG at quality factor 90 is indistinguishable from the original. At quality 60 you notice it a little, but it is still “the same photo”. At quality 30 the 8×8 blocks appear, the haloes around edges, the banding in gradients. At quality 10 the image is destroyed: you can tell there was a face, but the details have disappeared under the artefacts. And here is the key: no JPEG encoder exists that can compress to quality 10 without those artefacts, because the JPEG algorithm (blockwise DCT + coefficient quantisation) has an information floor below which its own mechanism introduces the noise.

What do you do if you need the photo at that file size and you need it to still look good? You do not compress the original harder. You repaint the photo knowing in advance that it is going to live compressed: an illustrator redraws it with clean lines, a reduced palette, no subtle gradients, an image designed to survive brutal compression. The result at “10 KB” looks infinitely better than the original JPEG crushed to 10 KB, because it is not the same process: one destroys existing information, the other generates new information already adapted to the constraint.

That is exactly the frontier of this post:

Scalar PTQ (≥4-bit) = compressing the JPEG. Up to a certain ratio, it is still the same photo.
State-of-the-art vector PTQ (2-bit: AQLM, QuIP#, QTIP) = a far more sophisticated image codec (dictionaries, decorrelating transforms) that stretches the compressible ratio a bit further before the collapse.
Native ternary (BitNet b1.58) = repainting the photo. You are not compressing an existing BF16 model; you are training a new one that is born ternary.

The map of the frontier, bit by bit

Quantising a model is deciding how many distinct values each weight can take. With b bits per weight there are 2^b possible values. The central question is: past what b is the number of values so small that rounding destroys the model?

Bits	Values/weight	State of the art	Method required	Typical loss vs BF16
8	256	Solved	RTN, SmoothQuant, FP8	~0 (indistinguishable)
4	16	Solved	AWQ, GPTQ	1-2 pp MMLU, +0.1-0.3 PPL
3	8	Small degradation	tuned GPTQ/AWQ, GGUF Q3_K	3-5 pp MMLU
2	4	Serious unless SOTA	AQLM, QuIP#, QTIP (non-scalar)	scalar: collapse; SOTA: 4-8 pp
1.58	3 (ternary)	Native only	BitNet b1.58 (QAT/native training)	n/a (not PTQ)
1	2 (binary)	Research	native, dubious claims	large / no clear methodology

The three transitions that matter:

8 → 4 bits: nothing breaks. With 16 levels per weight and one scale per block of 128, the rounding error is small relative to the dynamics of the weights. GPTQ compensates for the error by propagating it to the neighbouring weights; AWQ protects the ~1 % of salient channels. The model barely notices. This is in the previous post.

4 → 2 bits: the elbow. Two things happen at once here. First, with only 4 levels, the scalar quantiser can no longer represent the weight distribution, which is roughly Gaussian with long tails, without a proportionally enormous rounding error. Second, and more subtly: the quantisation error stops being “small noise that the model absorbs” and becomes structured, systematically biasing the activations. Naive scalar PTQ at 2 bits on a Llama 8B typically doubles perplexity or worse. That is the elbow of the curve.

2 → 1.58 bits: a change of nature. You do not cross this with a better compression method. You cross it by training the model with the constraint from the start. It is a discontinuity: to the left you are doing PTQ, to the right you are doing training.

Why scalar PTQ collapses below 4 bits

The scalar quantiser has a fundamental limitation: it quantises each weight separately, ignoring the fact that the weights of a row or column are correlated and that the error of one could be compensated by another. At 4 bits this matters little; at 2 bits it is lethal. There are three possible attacks, and the state-of-the-art methods use all three.

1. Vector quantisation: dictionaries instead of scales

Instead of mapping each weight to one of 4 values, group the weights into vectors (say, of 8 weights) and map each vector to the nearest entry of a learned dictionary (codebook). If the dictionary has 256 entries, encoding a vector of 8 weights costs 8 bits (the index) → 1 bit/weight, but every “reconstructed value” is a point in an 8-dimensional space chosen to minimise the error over the real distribution of the weights.

The advantage is information-theoretic: a vector dictionary can place its reconstruction points where the weights actually are (in clusters), whereas the scalar quantiser is forced to put its 4 levels on a regular grid, wasting resolution on empty regions. It is the difference between a road map with a uniform grid and one that puts more detail where the cities are.

AQLM (Additive Quantization of Language Models, arXiv:2401.06118) takes this to the extreme with additive quantisation: each weight vector is reconstructed as the sum of several codes from several dictionaries (multi-codebook). It is more expressive than a single dictionary because the number of combinations is the product of the sizes, not the sum. AQLM was one of the first methods to make 2-bit “usable” (not collapsed) on large models, at the cost of an expensive calibration process and specialised inference kernels.

2. Incoherence processing: whitening the matrix

The second attack targets the outliers. The weight matrices of a transformer have a few entries (and a few channels) with a magnitude much larger than the rest. Those outliers dominate the quantiser’s range: if you have to represent a weight of magnitude 8 while the rest are of magnitude 0.5, your scale stretches to cover the 8 and you waste almost all the resolution.

Incoherence processing (the core idea of QuIP and QuIP#) attacks this by multiplying the weight matrix W by random orthogonal matrices on the left and on the right: W' = U W V^T. Because U and V are orthogonal, the operation is invertible and the maths of the product can be undone at inference time by absorbing it into the neighbouring layers (just as AWQ absorbs its scales). But the rotation spreads the energy: an “incoherent” matrix has its values distributed almost uniformly, with no concentrated outliers, because mixing coordinates with a random rotation flattens the distribution (it is, in essence, the central limit theorem acting on linear combinations). A matrix without outliers quantises far better at 2 bits. It is the equivalent of “whitening” a signal before digitising it.

QuIP# (arXiv:2402.04396) combines incoherence processing with E8 lattice codes: rather than an arbitrary dictionary, it uses the E8 lattice (an optimal sphere packing in 8 dimensions, the best known). Quantising vectors of 8 weights against the E8 lattice gives the smallest reconstruction error possible for a given bit density, because E8 is literally the most efficient way to place points in 8D. It is classical coding theory applied to LLM weights.

3. Coding with memory: trellis

QTIP (arXiv:2406.11235) adds the third attack: trellis-coded quantisation. Instead of quantising each vector independently, it models the sequence of weights as a path through a trellis (the same structure as the convolutional codes of telecommunications) and picks the optimal code sequence with the Viterbi algorithm. The intuition: introducing memory between successive quantisations allows correlated errors that cancel out, instead of independent errors that accumulate. QTIP, on top of incoherence processing, improves on QuIP# in quality at 2-3 bits while keeping fast inference kernels.

The idea common to all three: stop quantising scalars and start quantising vectors with dictionaries, and decorrelate the matrix before doing so. None of them is “rounding better”; all three change the representation at the root. That is why, below 4 bits, a flag in vLLM is no longer enough: you need co-design of the quantisation method and the inference kernel.

Native ternary: BitNet b1.58

Here we change continents. Everything above is PTQ: it starts from a trained BF16 model and compresses it. BitNet’s ternary compresses nothing.

BitNet b1.58 (arXiv:2402.17764) trains a transformer from scratch where every weight is constrained to {-1, 0, +1} throughout training. Three values ⇒ log₂(3) ≈ 1.58 bits/weight. Quantisation is not a later step: the linear layers (BitLinear) quantise their weights to ternary in the forward pass of every training step, and gradients flow through a straight-through estimator. The model learns to work with ternary weights. This is QAT taken to the extreme: not a short robustness fine-tune, but the constraint present from the first token of training.

That difference is what dodges the elbow of the curve. PTQ at 2 bits tries to find the best ternary or quaternary approximation of a model that was trained expecting full precision, and that model has “fragile” weights that depend on nuances 2 bits cannot capture. BitNet, by contrast, never had those nuances: its weights were born ternary, so the network distributed its representational capacity in a way compatible with the constraint. It is repainting the photo instead of compressing it.

What changes is not just the memory, it is the arithmetic

The most underrated point about BitNet: with weights in {-1, 0, +1}, multiplication disappears from the matmul. Multiplying an activation x by a ternary weight w is trivial: if w = +1 you add x, if w = -1 you subtract x, if w = 0 you do nothing. The dominant operation of a transformer, the matrix-vector product, goes from a sea of floating point multiply-accumulate (MAC) to integer additions and subtractions.

This matters because it connects to the roofline. As explained in The inverted roofline of small models, LLM inference has two ceilings: memory (HBM bandwidth for loading weights) and compute (tensor core FLOPs). Normal quantisation (INT4, FP8) attacks only the memory ceiling: the weight takes up less space, but to multiply it you dequantise to FP16 and do the same multiplication as always. Ternary attacks both ceilings at once: the weight occupies 1.58 bits (memory) and the operation is an addition instead of a multiplication (compute). That is why BitNet needs its own kernels, bitnet.cpp, that execute the ternary matmul without ever going through FP16; a kernel that dequantised to FP16 in order to multiply would throw away half the advantage.

The honest trade-off: BitNet b1.58 is training from scratch. You cannot “convert your Llama 8B to BitNet”. If you want ternary, you train (or use) a natively ternary model, with everything that implies in pre-training cost and weight availability. Today it is a research line with published models at modest scales, not a drop-in replacement for your current serving.

QAT as the bridge between PTQ and native

Between “compressing post-hoc” (PTQ) and “training natively ternary” (BitNet) there is a middle point: QAT (Quantization-Aware Training). You take an already trained model and do a short fine-tune with the quantisation operations inside the loop, so that it learns to be robust to low bit counts without paying for a full pre-training.

Gemma 3 publishes official QAT variants for exactly this: models that, after the QAT fine-tune, hold up at INT4 with far less quality loss than pure PTQ on the same model. The cost is a training cost (GPU hours to days on an existing model), not an inference cost. For INT4 with QAT you recover almost all the quality; for 2-bit, QAT helps but the terrain is still difficult; for ternary, QAT stops being a “short fine-tune” and becomes native training (BitNet).

The decision hierarchy:

PTQ = default at ≥4 bits. Minutes to hours, without touching training weights. It covers 90 % of production.
QAT = when PTQ loses too much and the difference matters. Low bits (2-3), or sensitive models. You pay for a fine-tune.
Native (ternary) = when you want to go below 2 bits and change the arithmetic. You pay for pre-training. It only makes sense if you control the model from its creation.

The maths that matter: footprint and how many fit

The weight footprint is direct: bytes = (bits/param / 8) × N, with N the number of parameters. For an 8B model:

Level	bits/param	8B footprint	Ratio vs BF16
BF16	16	16.0 GB	1.0×
INT8	8	8.0 GB	2.0×
INT4	4	4.0 GB	4.0×
3-bit	3	3.0 GB	5.3×
2-bit	2	2.0 GB	8.0×
1.58-bit (ternary)	~1.58	~1.6 GB	~10×

(Real ternary takes up slightly more than 1.58 bits/param because you have to pack 5 ternary values into 8 bits, 5 × log₂(3) ≈ 7.92 bits, and because norms and embeddings usually stay at higher precision. The ~1.6 GB figure for 8B is the right order of magnitude.)

How many 8B models fit on an RTX 4090?

An RTX 4090 (24 GB, Ada Lovelace) has 24 GB. Reserving ~4 GB for KV cache and activations leaves 20 GB for weights. How many distinct 8B models fit loaded at the same time:

Level	8B footprint	Models in 20 GB	Comment
BF16	16.0 GB	1	one, with scant headroom left
INT8	8.0 GB	2	two different models
INT4	4.0 GB	5	the solved regime; quality ~lossless with AWQ
3-bit	3.0 GB	6	small degradation already visible
2-bit	2.0 GB	10	only viable with AQLM/QuIP#/QTIP
1.58-bit	~1.6 GB	~12	natively ternary models only

The arithmetic is seductive, from 1 to 12 models on the same card, but it should be read sceptically. Jumping from INT4 (5 models, almost lossless) to 2-bit (10 models) doubles the capacity, but only if you use a state-of-the-art method and accept 4-8 MMLU points. And the jump from 2-bit to ternary (10 → 12) is marginal in memory terms: ternary is not justified by footprint against a state-of-the-art 2-bit, it is justified by the arithmetic (the compute ceiling) and because being native it avoids the quality elbow. If your only metric is “how many GB it takes”, state-of-the-art 2-bit already gives you nearly everything. Ternary is for when you also want the compute saving.

The conceptual curve: perplexity vs bits

Three readings of the curve. One: to the right of 4 bits, the three branches are almost stuck together and almost flat, the solved regime. Two: the red branch (naive scalar PTQ) has a brutal elbow between 3 and 2 bits; that is where it doubles perplexity. The blue branch (state-of-the-art vector PTQ) flattens that elbow, it does not remove it, but it makes it tolerable down to 2 bits. Three: the green point of native ternary is on neither of the two PTQ curves, because it is not obtained by compressing: it is obtained by training, and that is why it can fall below the elbow without paying the quality price that any PTQ pays at that bit density. It is the difference between the crushed JPEG and the repainted photo.

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Three warnings for reading the aggressive-quantisation literature:

“1-bit lossless” almost always comes with small print. Pure binary {-1, +1} (1 bit) loses the ability to represent zero, which matters in transformers (many weights are effectively null). That is why the true state of the art at low density is ternary (1.58 bits), not binary: the zero is worth its extra 0.58 of a bit. When a paper announces “1-bit”, it pays to check whether (a) it really is 1 bit or 1.58 rounded down in the headline, (b) “lossless” is measured in WikiText perplexity (easy) or on reasoning benchmarks (where the collapse shows up), and (c) it compares against a baseline of the same effective size or against a much larger model in order to inflate the advantage.

Flat perplexity ≠ preserved quality. Perplexity on a generic corpus is the metric most forgiving of aggressive quantisation. A 2-bit model can have a perplexity almost identical to BF16 and at the same time drop 10 points on GSM8K or on a code benchmark, because multi-step reasoning amplifies errors that average perplexity does not see. Distrust any sub-4-bit claim that only reports perplexity. As we already said in the quantization post, the loss has to be measured on the target task.

Hardware comparability. The “X times faster” numbers for ternary only apply with the specialised kernels (bitnet.cpp) and on hardware where addition/subtraction arithmetic genuinely wins. On a GPU with tensor cores designed for FP16/FP8, a naive ternary kernel can be slower than a well-optimised INT4, because it wastes the silicon. The ternary advantage is real, but it is a co-design advantage (model + kernel + sometimes hardware), not a flag you switch on over your current stack. Any benchmark that does not specify the exact kernel and hardware is noise.

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace): the practical regime today is still INT4 AWQ for 7-14B models, solved, almost lossless, natively supported. State-of-the-art 2-bit (AQLM/QuIP#/QTIP) is viable and lets you load larger models or more models at once, but it demands the specific kernels of each method and an expensive calibration, and it pays in quality. It makes sense when VRAM is the bottleneck and you accept the trade-off; not as a default. Ternary on a 4090 is experimental: without tensor cores designed for ternary addition and subtraction, the compute advantage is diluted, although the memory saving remains.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8): here the default is FP8 (almost indistinguishable quality, native throughput) or INT4 AWQ for models that do not fit in FP8. State-of-the-art sub-4-bit is for serving enormous models (200B+) when neither FP8 nor INT4 fits with the KV cache headroom you want, at the cost of quality and kernel complexity. Native ternary, today, is a research object rather than a production one: its promise, hitting both roofline ceilings, is greater on CPU/edge (where there are no FP8 tensor cores to exploit) than on an H100 cluster, which already has dedicated FP8 hardware.

The rule of thumb, June 2026: ≥4-bit is settled engineering; state-of-the-art 2-bit is a real lever but with a cost in method and in quality; ternary is an architectural bet, not a deployment tweak.

References

Ma, S. et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58). https://arxiv.org/abs/2402.17764
Egiazarian, V., Panferov, A., Kuznedelev, D. et al. Extreme Compression of Large Language Models via Additive Quantization (AQLM). https://arxiv.org/abs/2401.06118
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., De Sa, C. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. https://arxiv.org/abs/2402.04396
Tseng, A., Sun, Q., Hou, D., De Sa, C. QTIP: Quantization with Trellises and Incoherence Processing. https://arxiv.org/abs/2406.11235
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
Google DeepMind. Gemma 3 QAT (Quantization-Aware Training) models — official blog: https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/
Microsoft. bitnet.cpp — 1-bit ternary inference kernels: https://github.com/microsoft/BitNet

Test-time quantization: quantising on the fly with no calibration dataset

Tue, 09 Jun 2026 02:00:00 +0000

This post is the natural continuation of Quantization for LLM inference, which is worth reading first: that is where GPTQ, AWQ, scale + zero-point and why activation outliers are the central problem all live. Here we are not discussing how many bits to use, but when and with what information the scales are computed: offline against a corpus (PTQ) or on the fly against real traffic (TTQ).

TL;DR

Activation-aware quantisation (AWQ, SmoothQuant) decides which channels to protect by measuring the magnitude of the activations over a calibration dataset in an offline pass, before deployment. The implicit assumption is that this corpus represents future traffic. But activation outliers, the channels with 10-100× the median magnitude that dominate the quantisation error, depend on the input: they change with the domain, the language and the client’s distribution. When real traffic drifts away from the calibration, the fixed scales stop being optimal and quality drops. Test-time quantization (TTQ) removes the corpus and the offline pass: it derives the activation-aware scales at inference time, from the activations that are actually observed, per token or per batch. The trade-off is honest and not minor: it introduces runtime overhead, computing statistics, detecting outliers, recomputing scales at every step, which competes directly with the saving from quantising. On small models that overhead weighs proportionally more, because the forward is short and the fixed per-step costs dominate (the framing is in the inverted roofline for SLMs). TTQ is orthogonal to the format: it is not a competitor to INT4 or FP8, it is a different way of deriving s. It pays off when there is no calibration pipeline, when the traffic distribution is shifting or unknown, and in multitenant settings where no representative corpus exists.

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

A clothes shop has two ways of dressing a customer.

The first is to sell off-the-peg sizes. At some point the factory measured an “average customer”, a mean mannequin built from a population sample, and cut the garments to those measurements. When a customer walks in, you hand them the size that comes closest. It is extremely fast: the garment is already sewn, it just gets handed over. The problem appears when the customer does not look like the average mannequin: if their shoulders are much broader than average, their particular outlier, the standard size pulls or leaves spare fabric, because it was cut to protect other areas. This is offline calibrated PTQ: AWQ measured the importance of each channel over a corpus and fixed the scales once and for all; fast at inference time, but blind to the specific customer.

The second is the tailor who takes measurements on the spot. When the customer walks in, the tailor gets out the tape, measures that customer, spots where their particular bulk is and adjusts the cut to their real anatomy. The result fits better, especially for customers who fall outside the mould. But every customer costs time: measuring, marking, deciding. This is TTQ: the scales are derived on the fly from the activations that the input actually generates.

The analogy holds up in three details:

The average mannequin = the calibration dataset. If the population walking into the shop resembles the mannequin, the sizes work; if not, they fail at the extremes.
Measuring every customer = computing activation statistics per token/batch. A better fit, but a fixed cost paid on every garment.
The broad shoulders = the outlier activation channels. They are precisely the areas where the fit matters and where the generic size gets it most wrong.

The tailor wins when the customers are varied or unknown. The tailor loses when you have a homogeneous population and a mannequin that represents it well: there, paying for measurement on every customer is a waste of time.

The problem TTQ solves: fixed calibration ages with the traffic

Let us recall from the quantization post what AWQ and SmoothQuant do exactly. They do not quantise all channels equally: they identify the ~1 % of channels whose activations have a large magnitude, the salient channels, and protect them by scaling them before quantising. To measure that importance they need to see activations, and they see them over a calibration dataset (128-512 samples, typically WikiText or a slice of the domain) in an offline pass prior to deployment.

The assumption is strong: that the activation distribution of the calibration corpus represents that of production traffic. Two reasons why that assumption breaks:

Activation outliers depend on the input. They are not a fixed property of the model the way weights are. The channel that is an outlier when processing C++ code may not be one when processing conversational Arabic or log JSON. The magnitude and position of the peaks change with the domain, the language and the input format.
Real traffic is rarely the corpus. You calibrate with English WikiText and the client sends you support tickets in Spanish with tables pasted in. The calibration protected the channels that WikiText activated, not the ones that real traffic activates. The scales are suboptimal exactly where the client lives.

The result is distribution-dependent degradation: the quantised model keeps its quality while the input resembles the calibration and loses it as the input drifts away. The most awkward case is multitenant: if you serve clients with different domains from the same quantised model, no single representative corpus exists; any fixed calibration favours some tenants and penalises others.

The TTQ mechanism: measure the real activations and scale on the fly

TTQ (arXiv:2603.19296, March 2026) proposes deriving activation-aware quantisation at inference time, with no offline pass and no calibration dataset. The idea, in its bare conceptual form:

Step 1 — Observe. When the activation tensor X arrives at a linear layer (per token or per batch), cheap statistics are computed over the channels: a measure of central tendency (median or mean magnitude) and one of dispersion per channel. This is the equivalent of AWQ looking at its corpus, but done over the activations that really are coming in right now.

Step 2 — Detect outliers on the fly. With those statistics, the channels whose magnitude shoots up relative to the tensor median are identified; the typical criterion is a threshold of the form “magnitude > k × median”. These are the channels that, if quantised with the same scale as the rest, blow up the error.

Step 3 — Derive scales and segregate. For normal channels a scale is computed that exploits the range; for the outliers a different treatment is applied, a scale of their own, or keeping them at higher precision, in the style of on-the-fly mixed precision. It is the same philosophy as LLM.int8() (segregating outliers to FP16) or AWQ (scaling salient channels), but with the threshold and the scales recomputed over the current input, not frozen since calibration.

Step 4 — Quantise and multiply. With the fresh scales, the tensor is quantised and the GEMM runs. The activations entering the next layer compensate for the rescaling, just as in AWQ, so that the maths cancels out.

The key difference with AWQ is not in what is done (protecting activation outliers) but in when and against what: AWQ decides once, offline, against a corpus; TTQ decides at every step, on the fly, against real traffic. It is the transfer to inference of the “test-time” idea: adapting the computation to the specific sample in front of you instead of to a precomputed average.

The maths that matter

The error of quantising an outlier with the wrong scale

Let us recall affine uniform quantisation from the base post: an integer code q = round(x/s) - z with scale s and zero-point z, and reconstruction x̂ = s·(q + z). For a b-bit quantiser with a symmetric range, the scale that covers a tensor of maximum magnitude M is approximately s = M / (2^{b-1} - 1). The rounding error of each element is bounded by half a scale: |x - x̂| ≤ s/2.

Here is the outlier problem. The scale s is chosen to cover the largest value in the group. If one channel has 30× the median magnitude and you share a single scale with the rest of the tensor, that magnitude rules: M is the outlier, so s is inflated 30× relative to what the majority would need. The absolute rounding error of the normal values rises proportionally.

A concrete calculation. Take a group where the median magnitude is 1.0 and an outlier channel is 30.0, quantised to INT4 (b = 4, levels ±7):

With a shared scale, s = 30 / 7 ≈ 4.29. The rounding error of a typical value (magnitude ~1) is up to s/2 ≈ 2.14. In other words, the error on the normal values is of the order of their own value: the outlier has destroyed the resolution of everything else. Relative error of a value of magnitude 1: up to ~214 %.
Segregating the outlier (pulling it out to FP16 or giving it its own scale) and quantising the rest with M = 1, s = 1/7 ≈ 0.143. The error of a typical value drops to s/2 ≈ 0.071, ~7 % relative. Thirty times less error on the majority of the group’s weights.

That is the entire reason for the existence of activation-aware quantisation: to detect and treat separately the ~1 % of channels that, if not segregated, hijack the scale. AWQ does it against the corpus; TTQ does it against the real input. And if the channel that is an outlier in production was not an outlier during calibration, AWQ did not protect it: it quantised real traffic with the inflated scale from the case above. That is where TTQ gains precision.

The overhead: the cost of measuring at every step

The price is symmetric. Computing the per-token statistics, per-channel magnitudes, median or percentile, outlier threshold, scales, means reductions over the activation tensor that did not exist in the forward pass with frozen scales. Let us call:

T = forward time per token with fixed scales (static PTQ), in µs.
Δ = extra per-token cost of deriving the statistics and scales on the fly, in µs.

The relative overhead is simply:

$$\text{overhead} = \frac{\Delta}{T}$$

The key point is that Δ is relatively fixed per step (it depends on the number of channels and layers, not on how much “useful” work the model does), while T scales with the size of the model. That is why the ratio behaves very differently depending on the model:

Large model (say, 70B): T is large, every forward moves tens of GB of weights from HBM. If Δ ≈ 8 µs and T ≈ 800 µs, the overhead is 8/800 = 1 %. Negligible against the saving from quantising.
SLM (say, 1B): T is small, the forward per token is short. With the same Δ ≈ 8 µs and T ≈ 60 µs, the overhead is 8/60 ≈ 13 %. No longer negligible: it eats a good part of what you gained by quantising.

This connects directly to the inverted roofline for small models: on an SLM the fixed per-step costs (kernel launches, synchronisations, overheads that do not scale with the model) weigh proportionally more, because there is less useful work to spread them over. TTQ’s Δ is exactly one of those fixed costs. Going per-batch instead of per-token amortises Δ across all the tokens of the batch and lowers the relative overhead, at the cost of coarser scales; it is the first parameter to touch.

The uncomfortable conclusion: TTQ gives away robustness to distribution but spends part of the acceleration budget on measuring, and in the regime where acceleration is scarcest, the SLMs, the ones most often deployed at the edge, is where that spending hurts most. It is not free; it is a change of currency.

A note of methodological scepticism: arXiv:2603.19296 is from March 2026, very recent, and as of the date of this post there are no broad independent reproductions. Whatever speedup and quality figures circulate should be taken with the same caution as any number without published methodology: what hardware, what batch size, what real measured Δ, against what baseline (well-calibrated or badly calibrated PTQ), in what domain? The conceptual argument, robustness to distribution in exchange for per-step overhead, is solid; the concrete multipliers are pending validation.

What TTQ is NOT: marking it off from the rest of the zoo

TTQ is easily confused with neighbouring techniques. The distinction that matters is that TTQ is the how of deriving the scales, not the format nor the moment in training.

Technique	When the scales are fixed	Needs a calibration corpus	Touches training	Is it a format
Static PTQ (GPTQ, AWQ)	Offline, before deployment	Yes	No	No (it uses INT4/INT8)
QAT	During training	No (training data)	Yes (retrains)	No
FP8 end to end	At runtime, but simple per-tensor scales	Minimal / none	No	Yes (E4M3/E5M2)
TTQ	At runtime, activation-aware per token/batch	No	No	No (orthogonal to the format)

The four distinctions, one by one:

Against static PTQ (GPTQ/AWQ). Same goal (protecting outliers), the same possible format (INT4), but PTQ freezes the decisions offline against a corpus and TTQ recomputes them on the fly. TTQ is, in a sense, “AWQ without the calibration phase, paid for at runtime”.
Against QAT. QAT puts quantisation inside the training loop so that the model learns to be robust to it; it costs a retrain. TTQ does not touch training: it operates on an already trained model, at inference. They are attacks at opposite moments of the pipeline.
Against FP8 end to end. FP8 is a format with its own logarithmic range; its “dynamic scaling” computes a simple per-tensor scalar at runtime, but it does not do activation-aware per-channel outlier detection. TTQ could, conceptually, derive scales on the fly for an FP8 or INT4 quantiser: it is orthogonal to the format.
TTQ is orthogonal to the format. It decides how to obtain s, not how many bits you store q in. You can imagine “TTQ over INT4” or “TTQ over FP8”. What defines TTQ is the source of the scale, real activations on the fly, not the width of the code.

When it pays off (and when it does not)

TTQ is not a universal replacement for AWQ. It is a tool for a specific deployment profile. It pays off when:

You have no calibration pipeline. You want to deploy a quantised model now, without assembling the calibration dataset, running the offline pass or validating that the corpus represents the traffic. TTQ cuts out that whole phase: you load the model and serve.
The traffic distribution is shifting or unknown. An assistant that one day receives code and the next legal contracts in another language. No fixed calibration covers both well; on-the-fly adaptation follows the distribution without recalibrating.
Multitenant with no representative corpus. You serve the same model to clients with disparate domains. No single corpus represents all of them; any fixed calibration creates winners and losers among tenants. TTQ adjusts to each input, whichever tenant it comes from.

It does not pay off when:

You have a stable domain and a good calibration corpus. If your traffic is homogeneous and representative, offline AWQ gives you the same quality with zero runtime overhead. Paying Δ on every token to relearn what a corpus already captured is waste.
You serve SLMs with a tight latency SLA. This is exactly the case where Δ/T is high. If the model is small and TPOT matters, the overhead of measuring can wipe out the gain from quantising. Measure your real Δ before assuming it adds up.
The batch is large and compute-bound. With high concurrency the forward is no longer memory-bound and the cost of the extra reductions competes worse; at the very least, amortise Δ per batch.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The natural case for the 4090 is the SLM, Qwen 3 1.5B, Llama 3 8B AWQ-INT4, serving at low concurrency. That is precisely the regime where TTQ is riskiest: T per token is small and the 4090 has no accelerated native FP8 (we discussed this in the quantization post), so TTQ’s extra reductions run on CUDA cores competing for the same time. Here the question is not “does it improve quality?” but “does the overhead leave me an acceptable TPOT?”. If the traffic is homogeneous, offline AWQ wins on simplicity and latency. TTQ only justifies its Δ if the input distribution is genuinely unpredictable and the degradation from fixed calibration is measurable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here the calculation partly flips. With large models T is high and Δ/T drops to the low single-digit percentage range, so TTQ’s overhead is more digestible. The strong use case is multitenant: a cluster serving a large model to clients with heterogeneous domains, where no calibration corpus satisfies everyone. There, TTQ’s robustness to distribution has real value and the overhead is diluted in a big forward. Even so, on an H100 with native FP8, the baseline to beat is demanding: static FP8 barely loses quality (see the table in the quantization post) and costs nothing at runtime. TTQ has to show that its robustness gain on the outlier tenants outweighs what it gives away in overhead. With a paper from March 2026 and no reproductions, that demonstration is pending.

What we have not covered

The memory cost of the on-the-fly statistics: per-channel buffers, their impact on the footprint and on cache pressure.
Interaction with continuous batching: how scales are derived when a batch mixes requests from different domains in the same step.
TTQ + speculative decoding: whether the draft and the target derive scales on the fly separately, and how that affects the acceptance rate.
Numerical stability: what happens when a batch contains a one-off extreme outlier that inflates the scale for every token of that step.

References

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly (March 2026). https://arxiv.org/abs/2603.19296
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024). https://arxiv.org/abs/2306.00978
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023). https://arxiv.org/abs/2210.17323
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023). https://arxiv.org/abs/2211.10438
Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022). https://arxiv.org/abs/2208.07339

FP8 end to end: enable it, measure quality and decide with data

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

FP8 is the configuration change with the highest impact per unit of effort available on H100 and Ada Lovelace hardware. On an H100 it enables native FP8 tensor cores: +40-60% decode throughput and ×2 the VRAM available for the KV cache. On an RTX 4090 and an L40 the compute benefit is smaller, but the ×2 VRAM is real and translates directly into twice the concurrency. The risk is quality degradation, which on well-calibrated modern models is <0.5% on standard benchmarks but can be larger on formal reasoning. The correct workflow is not to switch it on and pray: it is to switch it on in staging, run the eval suite, correlate quality with throughput in OTel, and decide with data.

The analogy

A photographer who works with 35 mm negatives and moves to digital. Digital photos take up less space and are processed faster. But a low-resolution photo of a landscape can be indistinguishable from the high-resolution one to the human eye, while a low-resolution photo of text loses letters. Exactly the same trade-off applies to FP8: for tasks where numerical imprecision is averaged out over thousands of activations (conversation, summarisation, RAG), it is practically invisible. For tasks where a single wrong multiplication propagates into an incorrect answer (formal mathematics, critical code), it can be decisive.

The three layers of FP8 in vLLM

FP8 is not a single flag: it is three independent layers that are enabled separately and have different benefits.

Layer 1 — Model weights (--quantization fp8): The model weights are stored and computed in FP8 E4M3. The models have to be pre-quantised (available on HuggingFace with a -FP8 or -fp8 suffix) or quantised at load time with calibration. The benefit: the model takes up half the VRAM and the weight matmuls are 2× faster on an H100.

# Pre-quantised model (recommended for production)
vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8

# Or on-the-fly quantisation (no extra files, somewhat slower on the first tokens)
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --quantization fp8 \
 --kv-cache-dtype auto

Layer 2 — KV cache (--kv-cache-dtype fp8): The K and V tensors of the KV cache are stored in FP8 instead of BF16. It halves the size of the KV cache, doubling the number of tokens that fit in VRAM. It does not affect the model weights.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # dynamic calibration, mandatory to minimise degradation

Layer 3 — Activations (automatic on an H100): On Hopper GPUs, vLLM automatically enables FP8 for the intermediate activations when both previous layers are active. No extra flag is required.

Full production configuration:

vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8 \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --gpu-memory-utilization 0.92 \
 --max-model-len 16384

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

Metric	BF16 baseline	FP8 enabled	Delta
Decode throughput (tok/s, 70B, batch 32)	~1,800	~2,700	+50%
Model VRAM (70B)	140 GB	70 GB	−50%
KV cache VRAM available (on 4×H100)	180 GB	250 GB	+39%
Maximum concurrency (ctx 8K)	~22,500 tok	~31,250 tok	+39%

This is equivalent to an extra replica for free in terms of KV cache capacity.

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

Metric	BF16/Q4 baseline	FP8 KV cache added	Delta
Decode throughput (tok/s, 14B Q4)	~45	~47	+4%
KV cache VRAM available	15 GB	15 GB (same model)	—
Total cache tokens (ctx 8K)	~46,000	~92,000	+100%
Maximum concurrency (ctx 8K)	~5 users	~11 users	+120%

On Ada, the compute benefit is smaller (the FP8 tensor cores do not have the same width as on Hopper), but the ×2 in KV cache capacity is entirely real and translates into twice the possible concurrent users.

The correct workflow: enable, measure, decide

Enabling FP8 directly in production without validating quality is inadequate. The correct workflow has four steps.

Step 1: baseline in staging

Before enabling FP8, record the quality metrics of the current BF16 model. The most reproducible way is to run an eval suite over a fixed dataset and save the results:

# Install lm-evaluation-harness
pip install lm-eval

# BF16 baseline
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/baseline_bf16.json

Step 2: enable FP8 and run the same eval suite

# FP8
lm_eval --model vllm \
 --model_args pretrained=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8,quantization=fp8,kv_cache_dtype=fp8,calculate_kv_scales=true \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/fp8_full.json

Step 3: compute the degradation

# compare_eval.py
import json

with open("results/baseline_bf16.json") as f:
 baseline = json.load(f)
with open("results/fp8_full.json") as f:
 fp8 = json.load(f)

tasks = ["mmlu", "hellaswag", "gsm8k"]
print(f"{'Task':<15} {'BF16':>8} {'FP8':>8} {'Delta':>8} {'OK?':>6}")
print("-" * 50)
for task in tasks:
 b = baseline["results"][task]["acc,none"]
 f = fp8["results"][task]["acc,none"]
 delta = (f - b) / b * 100
 ok = "✓" if abs(delta) < 1.0 else "✗ REVIEW"
 print(f"{task:<15} {b:>8.3f} {f:>8.3f} {delta:>+7.1f}% {ok:>6}")

Decision thresholds documented in MLPerf Inference 2025:

< 0.5% degradation: enable in production without restrictions.
0.5% – 1.5%: enable with active quality monitoring via LLM-as-judge.
> 1.5%: investigate before enabling — a possible calibration problem or an incompatible model.

Step 4: domain eval with LLM-as-judge

Academic benchmarks measure what they measure. Your use case may be different. Adding 200 representative samples from your domain, evaluated by an LLM judge, closes the gap:

# domain_eval.py
from langfuse import Langfuse
from openai import OpenAI

client = Langfuse()
judge = OpenAI(base_url="http://judge-llm:8000/v1", api_key="token")

# Load the 200 curated production samples (prompt + expected answer)
samples = load_domain_samples("eval_dataset_200.json")

scores_bf16, scores_fp8 = [], []
for sample in samples:
 for model_type, endpoint in [("bf16", "http://staging-bf16:8000"), ("fp8", "http://staging-fp8:8000")]:
 response = call_model(endpoint, sample["prompt"])
 score = judge.chat.completions.create(
 model="Qwen/Qwen2.5-72B-Instruct",
 messages=[{
 "role": "user",
 "content": f"Rate this answer from 1 to 5 on accuracy and completeness.\nQuestion: {sample['prompt']}\nExpected answer: {sample['expected']}\nModel answer: {response}\n\nReply with a number from 1 to 5 only."
 }]
 ).choices[0].message.content.strip()

 if model_type == "bf16":
 scores_bf16.append(int(score))
 else:
 scores_fp8.append(int(score))

import numpy as np
print(f"Mean BF16 score: {np.mean(scores_bf16):.2f}")
print(f"Mean FP8 score: {np.mean(scores_fp8):.2f}")
print(f"Degradation: {(np.mean(scores_fp8)-np.mean(scores_bf16))/np.mean(scores_bf16)*100:.1f}%")

OTel + Langfuse correlation: the dashboard that decides

The moment of decision rests on a single dashboard with two signals on the same time axis:

Signal 1 — Throughput (Prometheus):

rate(vllm:generation_tokens_total[5m])

Signal 2 — Mean quality (Langfuse → Prometheus via exporter):

# If you have configured Langfuse with scores exported via OTel
langfuse_score_value{name="llm_judge_domain"}

The expected pattern after enabling FP8: throughput rises by 40-60% and quality stays within ±0.1 points. If quality drops by more than 0.3 points and stays low, there is a real problem.

# Alert: quality drops more than 0.2 points, sustained, after the change
ALERT FP8CalidadDegradada
 IF avg_over_time(langfuse_score_value{name="llm_judge_domain"}[30m])
 < (avg_over_time(langfuse_score_value{name="llm_judge_domain"}[1d] offset 2h) - 0.2)
 FOR 15m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Possible quality degradation after the FP8 configuration change" }

When NOT to enable FP8

FP8 is not always the right answer. The cases where degradation exceeds the acceptable threshold:

Formal mathematical reasoning: GSM8K and MATH are the benchmarks most sensitive to FP8. If your use case is solving mathematical problems or precise financial calculation, measure specifically on these benchmarks before enabling it.

Critical code with tests: numerical precision affects the probability of tokens at key positions in a function. The risk is not that the code “looks” bad, but that it passes superficial tests while having subtle bugs.

Very long contexts without --calculate-kv-scales: without dynamic scale calibration, the accumulated numerical error in the KV cache grows with the context. With --calculate-kv-scales active, the impact is minimal up to 32K tokens.

Small models (<7B): the FP8 conversion overhead can exceed the throughput benefit. The break-even point is around 7B parameters.

References

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Wed, 27 May 2026 11:30:00 +0200

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.

Quantization on lo0 — Tech Blog

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

TL;DR

The reference identity

Master table: levers × impact × cost × tool

Lever 1: quantisation

Mechanics

Quality cost

OSS tooling

Lever 2: continuous batching

Mechanics

OSS tooling

Lever 3: power capping

Mechanics

The power cap vs throughput curve (H100 SXM)

J/token formula with a power cap

Commands

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

Commands (manual DVFS)

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

MoE vs dense

Model selection table by use case

Lever 6: speculative decoding

Mechanics

The energy efficiency condition

OSS tooling

Lever 7: prefix caching and chunked prefill

Prefix caching

Chunked prefill (SARATHI)

Lever 8: scheduling to clean hours

The power cap vs throughput curve: the sweet spot

How to measure the effect and validate quality

Measurement stack

Measurement protocol per lever

Quality validation

Pareto decision table (energy / latency / quality)

Stacking levers: the combined effect

Reference hardware: TDP and power cap ranges

Executive summary by role

Sources

Aggressive quantisation (state of the art): from 4-bit to ternary

TL;DR

The analogy: the JPEG that cannot be compressed any further

The map of the frontier, bit by bit

Why scalar PTQ collapses below 4 bits

1. Vector quantisation: dictionaries instead of scales

2. Incoherence processing: whitening the matrix

3. Coding with memory: trellis

Native ternary: BitNet b1.58

What changes is not just the memory, it is the arithmetic

QAT as the bridge between PTQ and native

The maths that matter: footprint and how many fit

How many 8B models fit on an RTX 4090?

The conceptual curve: perplexity vs bits

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Implications for on-premise inference

See also

References

Test-time quantization: quantising on the fly with no calibration dataset

TL;DR

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

The problem TTQ solves: fixed calibration ages with the traffic

The TTQ mechanism: measure the real activations and scale on the fly

The maths that matter

The error of quantising an outlier with the wrong scale

The overhead: the cost of measuring at every step

What TTQ is NOT: marking it off from the rest of the zoo

When it pays off (and when it does not)

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

References

FP8 end to end: enable it, measure quality and decide with data

TL;DR

The analogy