FP8 end to end: enable it, measure quality and decide with data

TL;DR

FP8 is the configuration change with the highest impact per unit of effort available on H100 and Ada Lovelace hardware. On an H100 it enables native FP8 tensor cores: +40-60% decode throughput and ×2 the VRAM available for the KV cache. On an RTX 4090 and an L40 the compute benefit is smaller, but the ×2 VRAM is real and translates directly into twice the concurrency. The risk is quality degradation, which on well-calibrated modern models is <0.5% on standard benchmarks but can be larger on formal reasoning. The correct workflow is not to switch it on and pray: it is to switch it on in staging, run the eval suite, correlate quality with throughput in OTel, and decide with data.

The analogy

A photographer who works with 35 mm negatives and moves to digital. Digital photos take up less space and are processed faster. But a low-resolution photo of a landscape can be indistinguishable from the high-resolution one to the human eye, while a low-resolution photo of text loses letters. Exactly the same trade-off applies to FP8: for tasks where numerical imprecision is averaged out over thousands of activations (conversation, summarisation, RAG), it is practically invisible. For tasks where a single wrong multiplication propagates into an incorrect answer (formal mathematics, critical code), it can be decisive.

The three layers of FP8 in vLLM

FP8 is not a single flag: it is three independent layers that are enabled separately and have different benefits.

Layer 1 — Model weights (--quantization fp8): The model weights are stored and computed in FP8 E4M3. The models have to be pre-quantised (available on HuggingFace with a -FP8 or -fp8 suffix) or quantised at load time with calibration. The benefit: the model takes up half the VRAM and the weight matmuls are 2× faster on an H100.

# Pre-quantised model (recommended for production)
vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
  --quantization fp8

# Or on-the-fly quantisation (no extra files, somewhat slower on the first tokens)
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype auto

Layer 2 — KV cache (--kv-cache-dtype fp8): The K and V tensors of the KV cache are stored in FP8 instead of BF16. It halves the size of the KV cache, doubling the number of tokens that fit in VRAM. It does not affect the model weights.

vllm serve my-model \
  --kv-cache-dtype fp8 \
  --calculate-kv-scales   # dynamic calibration, mandatory to minimise degradation

Layer 3 — Activations (automatic on an H100): On Hopper GPUs, vLLM automatically enables FP8 for the intermediate activations when both previous layers are active. No extra flag is required.

Full production configuration:

vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --calculate-kv-scales \
  --gpu-memory-utilization 0.92 \
  --max-model-len 16384

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

This is equivalent to an extra replica for free in terms of KV cache capacity.

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

On Ada, the compute benefit is smaller (the FP8 tensor cores do not have the same width as on Hopper), but the ×2 in KV cache capacity is entirely real and translates into twice the possible concurrent users.

The correct workflow: enable, measure, decide

Enabling FP8 directly in production without validating quality is inadequate. The correct workflow has four steps.

Step 1: baseline in staging

Before enabling FP8, record the quality metrics of the current BF16 model. The most reproducible way is to run an eval suite over a fixed dataset and save the results:

# Install lm-evaluation-harness
pip install lm-eval

# BF16 baseline
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct,dtype=bfloat16 \
  --tasks mmlu,hellaswag,gsm8k \
  --num_fewshot 5 \
  --output_path ./results/baseline_bf16.json

Step 2: enable FP8 and run the same eval suite

# FP8
lm_eval --model vllm \
  --model_args pretrained=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8,quantization=fp8,kv_cache_dtype=fp8,calculate_kv_scales=true \
  --tasks mmlu,hellaswag,gsm8k \
  --num_fewshot 5 \
  --output_path ./results/fp8_full.json

Step 3: compute the degradation

# compare_eval.py
import json

with open("results/baseline_bf16.json") as f:
    baseline = json.load(f)
with open("results/fp8_full.json") as f:
    fp8 = json.load(f)

tasks = ["mmlu", "hellaswag", "gsm8k"]
print(f"{'Task':<15} {'BF16':>8} {'FP8':>8} {'Delta':>8} {'OK?':>6}")
print("-" * 50)
for task in tasks:
    b = baseline["results"][task]["acc,none"]
    f = fp8["results"][task]["acc,none"]
    delta = (f - b) / b * 100
    ok = "✓" if abs(delta) < 1.0 else "✗ REVIEW"
    print(f"{task:<15} {b:>8.3f} {f:>8.3f} {delta:>+7.1f}% {ok:>6}")

Decision thresholds documented in MLPerf Inference 2025:

• < 0.5% degradation: enable in production without restrictions.
• 0.5% – 1.5%: enable with active quality monitoring via LLM-as-judge.
• > 1.5%: investigate before enabling — a possible calibration problem or an incompatible model.

Step 4: domain eval with LLM-as-judge

Academic benchmarks measure what they measure. Your use case may be different. Adding 200 representative samples from your domain, evaluated by an LLM judge, closes the gap:

# domain_eval.py
from langfuse import Langfuse
from openai import OpenAI

client = Langfuse()
judge = OpenAI(base_url="http://judge-llm:8000/v1", api_key="token")

# Load the 200 curated production samples (prompt + expected answer)
samples = load_domain_samples("eval_dataset_200.json")

scores_bf16, scores_fp8 = [], []
for sample in samples:
    for model_type, endpoint in [("bf16", "http://staging-bf16:8000"), ("fp8", "http://staging-fp8:8000")]:
        response = call_model(endpoint, sample["prompt"])
        score = judge.chat.completions.create(
            model="Qwen/Qwen2.5-72B-Instruct",
            messages=[{
                "role": "user",
                "content": f"Rate this answer from 1 to 5 on accuracy and completeness.\nQuestion: {sample['prompt']}\nExpected answer: {sample['expected']}\nModel answer: {response}\n\nReply with a number from 1 to 5 only."
            }]
        ).choices[0].message.content.strip()
        
        if model_type == "bf16":
            scores_bf16.append(int(score))
        else:
            scores_fp8.append(int(score))

import numpy as np
print(f"Mean BF16 score: {np.mean(scores_bf16):.2f}")
print(f"Mean FP8 score:  {np.mean(scores_fp8):.2f}")
print(f"Degradation: {(np.mean(scores_fp8)-np.mean(scores_bf16))/np.mean(scores_bf16)*100:.1f}%")

OTel + Langfuse correlation: the dashboard that decides

The moment of decision rests on a single dashboard with two signals on the same time axis:

Signal 1 — Throughput (Prometheus):

rate(vllm:generation_tokens_total[5m])

Signal 2 — Mean quality (Langfuse → Prometheus via exporter):

# If you have configured Langfuse with scores exported via OTel
langfuse_score_value{name="llm_judge_domain"}

The expected pattern after enabling FP8: throughput rises by 40-60% and quality stays within ±0.1 points. If quality drops by more than 0.3 points and stays low, there is a real problem.

# Alert: quality drops more than 0.2 points, sustained, after the change
ALERT FP8CalidadDegradada
  IF avg_over_time(langfuse_score_value{name="llm_judge_domain"}[30m]) 
     < (avg_over_time(langfuse_score_value{name="llm_judge_domain"}[1d] offset 2h) - 0.2)
  FOR 15m
  LABELS { severity = "warning" }
  ANNOTATIONS { summary = "Possible quality degradation after the FP8 configuration change" }

When NOT to enable FP8

FP8 is not always the right answer. The cases where degradation exceeds the acceptable threshold:

Formal mathematical reasoning: GSM8K and MATH are the benchmarks most sensitive to FP8. If your use case is solving mathematical problems or precise financial calculation, measure specifically on these benchmarks before enabling it.

Critical code with tests: numerical precision affects the probability of tokens at key positions in a function. The risk is not that the code “looks” bad, but that it passes superficial tests while having subtle bugs.

Very long contexts without --calculate-kv-scales: without dynamic scale calibration, the accumulated numerical error in the KV cache grows with the context. With --calculate-kv-scales active, the impact is minimal up to 32K tokens.

Small models (<7B): the FP8 conversion overhead can exceed the throughput benefit. The break-even point is around 7B parameters.

See also

• Multimodal on-premise: serving a VLM with vLLM (vision + language) — FP8 so that a VLM fits with its heavy budget of visual tokens.

• the maths of FP8 E4M3: what the 4-bit exponent and the 3-bit mantissa are, and why this specific format was chosen over INT8

• the KV cache size formula: why moving to FP8 divides it exactly by two

• --kv-cache-dtype fp8 and --calculate-kv-scales in the context of the full decode tuning

• how to set up the Langfuse + Prometheus correlation on a single dashboard for the FP8 before/after

• the full eval suite: how to build the 200-sample domain dataset and the LLM judge that verifies quality

In this same series

• the max-num-seqs × max-num-batched-tokens grid search: the free optimisation with the biggest impact before touching quantisation
• prefix cache hit rate engineering: going from 15% to 75% without adding hardware
• TP=4×1 vs TP=2×2: the architectural decision that determines how to scale what FP8 frees up

References

• The State of FP8 KV-Cache and Attention Quantization in vLLM — vLLM Blog (abril 2026)
• FP8 W8A8 — vLLM Documentation
• MLPerf Inference v5.1 — resultados de calidad FP8 (sep 2025)
• FP4 vs FP8 vs FP16 LLM Inference: Quality and Speed Tradeoffs
• vLLM Production Deployment 2026: FP8 Docker Setup on H100