GenAI-Perf in depth: LLM inference profiling with real data

Mon, 15 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N EUR), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

GenAI-Perf (the genai-perf package, part of NVIDIA’s Triton ecosystem) is the reference LLM inference profiler for OpenAI-compatible endpoints. Installable with pip install genai-perf. A single genai-perf profile command produces a console table with TTFT, ITL, request latency (avg/min/max/P75/P90/P99), output sequence length, output token throughput and request throughput, plus profile_export_genai_perf.json and .csv artefacts. The analyze subcommand automatically sweeps concurrencies from 1 to 256 (powers of 2) and generates a summary analyze_export_genai_perf.csv. GenAI-Perf was declared retired in April 2026 (replaced by AIPerf); it remains the reference benchmark for historical runs and documented comparisons, and the NVIDIA tool with the most public documentation for LLMs before 2026.

What GenAI-Perf is and where it fits

GenAI-Perf is a command-line tool for measuring the throughput and latency of generative AI models served through an inference server. It is part of the triton-inference-server/perf_analyzer GitHub repository and is distributed as a pip package (genai-perf) and inside the Triton SDK container (nvcr.io/nvidia/tritonserver:YY.MM-py3-sdk).

Position in the NVIDIA ecosystem:

Relationship with perf_analyzer: GenAI-Perf internally uses Triton’s perf_analyzer binary to generate load and measure latencies. perf_analyzer is the low-level load generator; GenAI-Perf is the high-level layer that adds prompt synthesis, OpenAI payload construction and the LLM-specific metrics (TTFT, ITL, sequence lengths).

Installation:

# Option 1: pip (requires CUDA 12, Ubuntu 24.04, Python 3.10+)
pip install genai-perf

# Option 2: Triton SDK container (recommended for reproducibility)
export RELEASE="25.01"
docker run -it --net=host --gpus=all \
 nvcr.io/nvidia/tritonserver:25.01-py3-sdk
# Inside the container:
genai-perf --help

Metrics GenAI-Perf produces

The metrics cover the full cycle of a streaming generation request. All of them are reported with avg, min, max, P75, P90 and P99 except the throughput ones (avg only).

Metrics glossary

Metric	Symbol	Exact definition (GenAI-Perf)	Unit
Time to First Token	TTFT	Time from sending the request to receiving the first token (includes queueing, prefill and network)	ms
Inter-Token Latency	ITL (= TPOT)	$\frac{e2e\_latency - TTFT}{output\_tokens - 1}$ — decode phase only, excludes the first token	ms/token
Request Latency	e2e_latency	$e2e = TTFT + Generation\_time$ — from the first request to the last response	ms
Output Token Throughput	OTT	$\frac{total\_output\_tokens}{T_y - T_x}$ where $T_x$ = first request, $T_y$ = last token received	tok/s
Request Throughput	RPS	$\frac{total\_completed\_requests}{T_y - T_x}$	req/s
Input Sequence Length	ISL	Mean length in tokens of the input prompt	tokens
Output Sequence Length	OSL	Mean length in tokens of the generated response	tokens

Methodological note on ITL versus LLMPerf: GenAI-Perf excludes TTFT from the ITL calculation (pure decode only). LLMPerf includes TTFT in its ITL mean. The numbers are not directly comparable.

Methodological note on TPS versus LLMPerf: GenAI-Perf measures throughput between the first request and the last token received ($T_y - T_x$). LLMPerf uses the total duration of the benchmark including prompt generation and response storage; in a concurrency-1 scenario, that difference can amount to up to 33% in the reported throughput figure.

Console output table (real example, TRT-LLM backend)

The table GenAI-Perf prints for a profile with ISL=200, OSL=100, concurrency=1:

 NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Statistic ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Time to first token (ms) │ 13.68 │ 11.07 │ 21.50 │ 18.81 │ 14.29 │ 13.97 │
│ Inter token latency (ms) │ 1.86 │ 1.28 │ 2.11 │ 2.11 │ 2.01 │ 1.95 │
│ Request latency (ms) │ 203.70 │ 180.33 │ 228.30 │ 225.45 │ 216.48 │ 211.72 │
│ Output sequence length │ 103.46 │ 95.00 │ 134.00 │ 122.96 │ 108.00 │ 104.75 │
│ Input sequence length │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │
│ Output token throughput (per sec) │ 504.02 │ N/A │ N/A │ N/A │ N/A │ N/A │
│ Request throughput (per sec) │ 4.87 │ N/A │ N/A │ N/A │ N/A │ N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

(Source: docs.nvidia.com/deeplearning/triton-inference-server — LLM tutorial)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

The profile subcommand measures a fixed operating point (concurrency or request-rate, synthetic ISL/OSL).

Example 1: vLLM endpoint (chat, streaming)

# 1. Bring up the vLLM server
docker run -it --net=host --rm --gpus=all \
 vllm/vllm-openai:latest \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --dtype float16

# 2. Profile with GenAI-Perf
genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --concurrency 16 \
 --synthetic-input-tokens-mean 1024 \
 --synthetic-input-tokens-stddev 128 \
 --output-tokens-mean 256 \
 --output-tokens-stddev 0 \
 --output-tokens-mean-deterministic \
 --request-count 200 \
 --warmup-request-count 20 \
 --tokenizer meta-llama/Llama-3.1-70B-Instruct

Example 2: completions endpoint (no streaming)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type completions \
 --concurrency 32 \
 --synthetic-input-tokens-mean 512 \
 --synthetic-input-tokens-stddev 64 \
 --output-tokens-mean 128 \
 --request-count 200 \
 --warmup-request-count 20

Example 3: NVIDIA NIM (TRT-LLM backend endpoint, via Triton)

genai-perf profile \
 -m meta/llama-3.1-70b-instruct \
 --backend tensorrtllm \
 --streaming \
 --concurrency 8 \
 --synthetic-input-tokens-mean 2048 \
 --synthetic-input-tokens-stddev 0 \
 --output-tokens-mean 512 \
 --output-tokens-mean-deterministic \
 --request-count 100 \
 --warmup-request-count 10 \
 --generate-plots

Example 4: request-rate instead of concurrency (Poisson)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --request-rate 5.0 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 300 \
 --warmup-request-count 30

Main flags and what they do

Flag	Typical values	Function
`--endpoint-type`	`chat`, `completions`, `embeddings`	Type of OpenAI-compatible endpoint
`--streaming`	(boolean)	Enables SSE streaming; required to measure real TTFT and ITL
`--concurrency`	1, 8, 16, 32, 64	Number of concurrent requests maintained; GenAI-Perf guarantees N active at all times
`--request-rate`	1.0, 5.0, 10.0	Constant arrival rate (req/s); does not guarantee N active
`--synthetic-input-tokens-mean`	128–8192	Mean of synthetic ISL tokens
`--synthetic-input-tokens-stddev`	0–512	Standard deviation of the ISL (0 = fixed)
`--output-tokens-mean`	64–2048	Mean of target OSL tokens
`--output-tokens-mean-deterministic`	(boolean)	Sets the minimum output tokens = target mean (more precise with TRT-LLM)
`--request-count`	100–2000	Number of requests to benchmark
`--warmup-request-count`	10–50	Warm-up requests discarded from the metrics
`--generate-plots`	(boolean)	Generates PNG plots of TTFT vs ISL, ITL vs token position, etc.
`--tokenizer`	HF model id	Tokenizer for counting tokens; mandatory when ISL/OSL matter
`--backend`	`tensorrtllm`, `vllm`	Triton backend (for serving directly via Triton without an OpenAI endpoint)
`--extra-inputs`	`ignore_eos:true`	Extra request parameters; `ignore_eos:true` guarantees a consistent OSL

Note on --concurrency versus --request-rate: NVIDIA recommends using --concurrency. With --request-rate, if the rate exceeds the engine’s throughput, the queue grows without bound and the metrics stop being stable. With --concurrency, there are always exactly N active requests, which gives a clean latency-throughput curve.

Output artefacts

GenAI-Perf dumps all results into an artifacts/ directory:

Artefact	Format	Content
`profile_export_genai_perf.json`	JSON	Full metrics (avg, min, max, P75, P90, P99) + CLI arguments used
`profile_export_genai_perf.csv`	CSV	Exported console tables, ready to import into Excel/pandas
`profile_export.json`	JSON	Raw perf_analyzer data (per-request traces)
`inputs.json`	JSON	Synthetic payloads sent (for reproducibility)
PNG plots	PNG	TTFT analysis, Request latency, TTFT vs ISL, ITL vs position, ISL vs OSL (with `--generate-plots`)

The `analyze` subcommand: automatic concurrency sweep

The analyze subcommand sweeps multiple values of a parameter in a single command and generates a summary CSV.

Concurrency sweep (the most used)

# Sweeps concurrencies 1, 2, 4, 8, 16, 32, 64, 128, 256
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type concurrency \
 --sweep-range 1:256 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Request-rate sweep

# Sweeps 2, 4, 6, 8, 10, 12 req/s
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type request_rate \
 --sweep-list 2,4,6,8,10,12 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

ISL sweep (to study the effect of prefill)

genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type input_sequence_length \
 --sweep-list 256,512,1024,2048,4096 \
 --concurrency 16 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Artefact structure of `analyze`

For a sweep with --sweep-type concurrency --sweep-range 1:32:

artifacts/
llama70b-openai-chat-concurrency1/
inputs.json
profile_export.json
profile_export_genai_perf.json
profile_export_genai_perf.csv
llama70b-openai-chat-concurrency2/
...
llama70b-openai-chat-concurrency4/
...
[...]
analyze_export_genai_perf.csv ← summary of every scenario
checkpoint.json ← allows interrupted sweeps to be resumed

Summary CSV format

Config Name,Concurrency,ISL,p99 TTFT (ms),p99 ITL (ms),p99 Request Latency (ms),Avg. OTT (tok/s),RPS
llama70b_run_0,1,1024,45.2,6.8,1823.4,128.3,0.55
llama70b_run_1,4,1024,48.1,7.1,1901.2,510.7,2.18
llama70b_run_2,8,1024,61.3,7.9,2142.5,989.2,4.12
llama70b_run_3,16,1024,98.7,9.4,2891.3,1741.6,6.88
llama70b_run_4,32,1024,284.1,14.2,4320.8,2134.2,7.14
llama70b_run_5,64,1024,892.4,28.7,8741.2,2251.8,7.23

The summary CSV also includes a second table with GPU metrics (P99 power, energy, utilisation, memory) when they are captured via Triton’s telemetry URLs (--server-metrics-urls http://localhost:8002/metrics).

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

The choice of tool changes the number. They are not comparable with each other without adjustment.

Dimension	GenAI-Perf	GuideLLM	LLMPerf	vLLM bench serve
Origin / maintainer	NVIDIA / Triton team (retired Apr 2026)	Red Hat / vLLM project	Anyscale / Ray	vLLM project
Class	multi-process load generator	multi-process load generator	multi-process load generator	single-process micro-bench
Supported endpoints	OpenAI-compatible + KServe + native Triton	OpenAI-compatible	OpenAI-compatible	native vLLM
Main load mode	fixed concurrency N (recommended) or constant request-rate	synchronous, concurrent, poisson, throughput, automatic sweep	batches of N concurrent (with a draining period at the end)	request-rate, concurrency
Draining period	NO — guarantees N active at all times	NO (poisson)	YES — at the end of each batch the system empties, concurrency drops to 0	N/A
Automatic sweep	`analyze` (concurrency, request-rate, ISL, OSL)	`--rate-type sweep` (idle to saturation, 10 rounds)	manual (several runs)	`vllm bench sweep serve`
LLM metrics	TTFT, ITL, e2e latency, OTT, RPS, ISL/OSL	TTFT, ITL/TPOT, throughput, goodput under SLO	TTFT, ITL (includes TTFT in the mean), TPS (total benchmark duration)	TTFT, TPOT, throughput
ITL difference	excludes TTFT: $(e2e - TTFT) / (N_{tok}-1)$	excludes TTFT (same as GenAI-Perf)	includes TTFT in the mean	excludes TTFT
TPS difference	$total\_tokens / (T_y - T_x)$	throughput + goodput under SLO	$total\_tokens / (T_{end} - T_{start})$ — up to 33% lower	$total\_tokens / (T_y - T_x)$
Goodput / SLO	NO	YES (the key differentiator)	NO	NO
Exports	JSON + CSV + PNG + checkpoint	JSON + YAML + CSV + interactive HTML	JSON	console / JSON
Built-in warm-up	YES (`--warmup-request-count`)	YES (warm-up requests)	NOT native	YES
Real datasets	OpenOrca, CNN DailyMail, JSONL file, moon_cake	file with a traffic distribution	JSONL file	sharegpt, random
GPU telemetry	YES (via Triton’s server-metrics-urls → summary CSV)	NOT native (integrate DCGM separately)	NO	NO
When to choose it	fine profiling of one operating point, comparing configs with the same harness, integration with NIM/Triton	SLO-driven sweep, finding the knee, sizing replicas	quick endpoint validation in the Ray ecosystem	iterating on vLLM flags

Fair-comparison rule: never cross numbers from different tools for the same conclusion. If vLLM and TRT-LLM are being compared, use GenAI-Perf against both, with the same ISL/OSL, the same concurrency and the same --warmup-request-count. The tool is the constant; the engine is the variable.

Profiling on 4×H100 SXM 80 GB: a full sweep example

Generic reference hardware: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP16, ISL=1024 tokens, OSL=256 tokens, vLLM endpoint.

Illustrative result of a concurrency sweep (1→64)

Concurrency	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	ITL P99 (ms)	OTT (tok/s)	RPS
1	38	45	6.4	7.1	130	0.51
4	40	51	6.8	8.2	510	2.00
8	46	62	7.3	9.4	980	3.83
16	68	98	8.9	12.1	1,720	6.72
32	145	284	13.7	21.4	2,130	8.32
48	380	740	22.3	38.6	2,280	8.90
64	890	1,820	41.2	78.3	2,340	9.14

How to read it: throughput saturates around concurrency 48–64, while TTFT P99 already exceeds 500 ms from concurrency 32 onwards. The knee (the last point with TTFT P99 < 500 ms) is at concurrency 16 with 1,720 useful tok/s. Anyone reporting “2,340 tok/s” (concurrency 64) is describing the maximum throughput, at a point where the system meets no interactive chat SLO at all.

Formulas for the key metrics

$$ \text{ITL} = \frac{e2e\_latency - TTFT}{N_{output\_tokens} - 1} $$ $$ \text{OTT} = \frac{\sum output\_tokens}{T_y - T_x} $$ $$ \text{RPS} = \frac{N_{requests}}{T_y - T_x} $$

where $T_x$ is the timestamp of the first request sent and $T_y$ the timestamp of the last token received from the last request.

Synthetic ISL/OSL profiling: what to configure for each use case

The ISL/OSL parameters determine which engine of the system is stressed: a long ISL stresses prefill (KV-cache, TTFT); a long OSL stresses decode (ITL, memory bandwidth).

Use case	Typical ISL	Typical OSL	`--synthetic-input-tokens-mean`	`--output-tokens-mean`
Short interactive chat	~300 tok	~100 tok	300	100
Code copilot	~800 tok	~200 tok	800	200
Document summarisation	~2000 tok	~256 tok	2000	256
RAG with long context	~4096 tok	~512 tok	4096	512
Batch without latency	~512 tok	~1024 tok	512	1024

The --extra-inputs ignore_eos:true option disables the engine’s EOS token, forcing exactly the requested tokens to be generated. Essential for measuring a consistent OSL on TRT-LLM; optional on vLLM.

Honest methodology: how not to measure badly

1. Warm-up is mandatory

Without warm-up, the first requests include the KV-cache loading latency, the engine’s autotuning and the initialisations. GenAI-Perf applies a sliding window to detect stability (max/min ratio within a margin over the last 3 measurements), but the explicit --warmup-request-count discards the first N requests from the metrics. Recommended minimum value: 10 requests (or 5% of the total if the total is large).

2. The concurrency knee: extend the sweep until you see it

Throughput saturates once the engine’s effective max batch size is reached. If the sweep stops before saturation is reached, the “maximum capacity” reported is the last concurrency tested, not the real knee. The rule: the sweep must reach a point where TTFT P99 has doubled relative to the low-load value (or where OTT stops growing by more than 5% between steps). Only then is the knee visible.

For a sweep with --sweep-range 1:256, GenAI-Perf sweeps 1, 2, 4, 8, 16, 32, 64, 128, 256 (powers of 2). On most 70B models over 4×H100, saturation occurs between concurrency 32 and 64; the range 1:128 covers the knee with room to spare.

3. Concurrency versus request-rate: choosing correctly

Mode	Behaviour	When to use
`--concurrency N`	Always N active requests; when one finishes, the next is sent	Clean latency-throughput curve; comparison between configs
`--request-rate R`	1 request is sent every 1/R seconds; the queue can grow without bound	Simulating traffic with arrivals at a constant rate

NVIDIA recommends --concurrency for benchmarking. With --request-rate, if R exceeds the engine’s maximum throughput, the queue grows indefinitely and the metrics become unstable.

4. Single-process versus multi-process bias

GenAI-Perf is multi-process: it maintains N concurrent requests using perf_analyzer’s internal manager. There is no risk of client saturation. This sets it apart from vllm bench serve (single-process), which can saturate on the client at high concurrency and give inflated throughput metrics.

5. `ignore_eos` and an inconsistent OSL

Without --extra-inputs ignore_eos:true, the engine may generate fewer tokens than requested (if the model produces an EOS earlier). This makes the real OSL vary between requests and the ITL become inconsistent between runs. For reproducible benchmarks with a controlled OSL: always enable ignore_eos:true.

6. The correct tokenizer

The --tokenizer flag must point to the model being served. If it is omitted, GenAI-Perf uses a default tokenizer that can give different ISL/OSL in the real model’s tokens, making the throughput numbers in tok/s not comparable between models with different vocabularies.

7. One run = a data point; three or more = a number

Variance between runs under identical conditions can be 5–10% on tail latency metrics (P99). A single profile is not a defensible number; the mean of three runs with the same config is.

Connection with the cost and energy track

GenAI-Perf can capture GPU metrics in the same summary CSV if --server-metrics-urls http://localhost:8002/metrics is passed (Triton’s metrics endpoint, which exposes power and utilisation via DCGM). The analyze summary CSV then includes a second table with P99 power (W), energy (MJ), utilisation (%) and memory (GB) per GPU and per scenario.

With that data, a single sweep yields the three axes of the scorecard:

Axis	Source	Metric
Performance	GenAI-Perf	OTT (tok/s), TTFT P99, ITL P99
Energy	DCGM via server-metrics-urls	Power (W) → $J/token = W / OTT$
Cost	node price	EUR/hour → $EUR/token = EUR_{hour} / (OTT \times 3600)$

The formula for cost per million tokens:

$$ CPM = \frac{EUR_{hour}}{OTT \times 3{.}6 \times 10^3} $$

where $OTT$ is the output token throughput in tok/s and $CPM$ is the cost per million tokens in EUR.

It connects with GPU observability with DCGM for the power capture, and with anatomy of an LLM request for the breakdown of phases (prefill versus decode) that TTFT and ITL quantify.

Status 2026: GenAI-Perf → AIPerf

NVIDIA announced in April 2026 that GenAI-Perf moves to passive maintenance mode (no new features) and that the active successor is AIPerf (github.com/ai-dynamo/aiperf). AIPerf adds automatic detection of the saturation point (estimatedCapacity) and is integrated into the NVIDIA Dynamo pipeline.

For new benchmarks in production: migrate to AIPerf. To reproduce historical runs documented with GenAI-Perf, or to use NVIDIA’s NIM benchmarking artefacts (which use GenAI-Perf): stay on GenAI-Perf 25.x.

The article LLM benchmark tools, one by one covers the full comparison including AIPerf. The article GuideLLM: SLO validation under load details the SLO-driven sweep that GenAI-Perf does not do natively. For the general methodology context: LLM benchmarking frameworks, state of the art.

Sources

NVIDIA · GenAI-Perf — README oficial — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · Tutorial LLM con GenAI-Perf (comandos y tablas de salida reales) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
NVIDIA · GenAI-Perf Analyze subcommand (sweep, CSV resumen, checkpoint) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/analyze.html
NVIDIA · NIM LLM Benchmarking — Métricas (definiciones TTFT, ITL, TPS, RPS, fórmulas exactas) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
NVIDIA · NIM LLM Benchmarking — Parámetros y buenas prácticas (ISL/OSL, concurrencia vs request-rate, ignore_eos) — https://docs.nvidia.com/nim/benchmarking/llm/latest/parameters.html
NVIDIA Technical Blog · Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API — https://developer.nvidia.com/blog/measuring-generative-ai-model-performance-using-nvidia-genai-perf-and-an-openai-compatible-api/
GitHub · triton-inference-server/perf_analyzer (genai-perf) — https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
GitHub · ray-project/llmperf (métricas y diferencias metodológicas) — https://github.com/ray-project/llmperf
PyPI · nvidia-genai-perf-eval — https://pypi.org/project/nvidia-genai-perf-eval/
Macnica · Benchmarking LLM Applications Part 1: What is GenAI-Perf? — https://www.macnica.co.jp/en/business/semiconductor/articles/nvidia/145977/

Genai-Perf on lo0 — Tech Blog

GenAI-Perf in depth: LLM inference profiling with real data

TL;DR

What GenAI-Perf is and where it fits

Metrics GenAI-Perf produces

Metrics glossary

Console output table (real example, TRT-LLM backend)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

Main flags and what they do

Output artefacts

The `analyze` subcommand: automatic concurrency sweep

Concurrency sweep (the most used)

Request-rate sweep

ISL sweep (to study the effect of prefill)

Artefact structure of `analyze`

Summary CSV format

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

Profiling on 4×H100 SXM 80 GB: a full sweep example

Illustrative result of a concurrency sweep (1→64)

Formulas for the key metrics

Synthetic ISL/OSL profiling: what to configure for each use case

Honest methodology: how not to measure badly

1. Warm-up is mandatory

2. The concurrency knee: extend the sweep until you see it

3. Concurrency versus request-rate: choosing correctly

4. Single-process versus multi-process bias

5. `ignore_eos` and an inconsistent OSL

6. The correct tokenizer

7. One run = a data point; three or more = a number

Connection with the cost and energy track

Status 2026: GenAI-Perf → AIPerf

See also

Sources

Genai-Perf on lo0 — Tech Blog

GenAI-Perf in depth: LLM inference profiling with real data

TL;DR

What GenAI-Perf is and where it fits

Metrics GenAI-Perf produces

Metrics glossary

Console output table (real example, TRT-LLM backend)

Invocation against OpenAI-compatible endpoints

profile mode: one load point

Main flags and what they do

Output artefacts

The analyze subcommand: automatic concurrency sweep

Concurrency sweep (the most used)

Request-rate sweep

ISL sweep (to study the effect of prefill)

Artefact structure of analyze

Summary CSV format

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

Profiling on 4×H100 SXM 80 GB: a full sweep example

Illustrative result of a concurrency sweep (1→64)

Formulas for the key metrics

Synthetic ISL/OSL profiling: what to configure for each use case

Honest methodology: how not to measure badly

1. Warm-up is mandatory

2. The concurrency knee: extend the sweep until you see it

3. Concurrency versus request-rate: choosing correctly

4. Single-process versus multi-process bias

5. ignore_eos and an inconsistent OSL

6. The correct tokenizer

7. One run = a data point; three or more = a number

Connection with the cost and energy track

Status 2026: GenAI-Perf → AIPerf

See also

Sources

`profile` mode: one load point

The `analyze` subcommand: automatic concurrency sweep

Artefact structure of `analyze`

5. `ignore_eos` and an inconsistent OSL