LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

where \(N_{\text{out}}\) is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific \(N_{\text{out}}\): both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Reading notes:

• “Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
• Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
• Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
• In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Harness steps

1. Deploy the engine with the exact config (pinned flags, fixed version).
2. Warm-up: 200 requests; discard the results.
3. Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
4. Extend the sweep past the knee (until goodput falls below 50 %).
5. Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
6. Change only the engine; repeat steps 1–5.
7. Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
  "harness": "aiperf", "harness_version": "2.x",
  "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
  "hardware": "4xH100_SXM_80GB_NVLink",
  "dataset": "sharegpt",
  "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
  "motor": "vllm", "motor_version": "0.9.x",
  "concurrency_sweep": [8, 16, 32, 48, 64],
  "results_at_knee": {
    "concurrency": 32,
    "ttft_p50_ms": 210, "ttft_p99_ms": 420,
    "itl_p50_ms": 18, "throughput_tok_s": 12500,
    "goodput_tok_s": 12200
  }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Sources

• vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
• vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
• vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
• SGLang · documentación oficial — https://docs.sglang.io/
• SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
• SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
• NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
• NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
• NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
• NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
• NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
• NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
• Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
• Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
• MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
• Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
• arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
• IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html