Nvidia on lo0 — Tech Blog

GenAI-Perf in depth: LLM inference profiling with real data

Mon, 15 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N EUR), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

GenAI-Perf (the genai-perf package, part of NVIDIA’s Triton ecosystem) is the reference LLM inference profiler for OpenAI-compatible endpoints. Installable with pip install genai-perf. A single genai-perf profile command produces a console table with TTFT, ITL, request latency (avg/min/max/P75/P90/P99), output sequence length, output token throughput and request throughput, plus profile_export_genai_perf.json and .csv artefacts. The analyze subcommand automatically sweeps concurrencies from 1 to 256 (powers of 2) and generates a summary analyze_export_genai_perf.csv. GenAI-Perf was declared retired in April 2026 (replaced by AIPerf); it remains the reference benchmark for historical runs and documented comparisons, and the NVIDIA tool with the most public documentation for LLMs before 2026.

What GenAI-Perf is and where it fits

GenAI-Perf is a command-line tool for measuring the throughput and latency of generative AI models served through an inference server. It is part of the triton-inference-server/perf_analyzer GitHub repository and is distributed as a pip package (genai-perf) and inside the Triton SDK container (nvcr.io/nvidia/tritonserver:YY.MM-py3-sdk).

Position in the NVIDIA ecosystem:

Relationship with perf_analyzer: GenAI-Perf internally uses Triton’s perf_analyzer binary to generate load and measure latencies. perf_analyzer is the low-level load generator; GenAI-Perf is the high-level layer that adds prompt synthesis, OpenAI payload construction and the LLM-specific metrics (TTFT, ITL, sequence lengths).

Installation:

# Option 1: pip (requires CUDA 12, Ubuntu 24.04, Python 3.10+)
pip install genai-perf

# Option 2: Triton SDK container (recommended for reproducibility)
export RELEASE="25.01"
docker run -it --net=host --gpus=all \
 nvcr.io/nvidia/tritonserver:25.01-py3-sdk
# Inside the container:
genai-perf --help

Metrics GenAI-Perf produces

The metrics cover the full cycle of a streaming generation request. All of them are reported with avg, min, max, P75, P90 and P99 except the throughput ones (avg only).

Metrics glossary

Metric	Symbol	Exact definition (GenAI-Perf)	Unit
Time to First Token	TTFT	Time from sending the request to receiving the first token (includes queueing, prefill and network)	ms
Inter-Token Latency	ITL (= TPOT)	$\frac{e2e\_latency - TTFT}{output\_tokens - 1}$ — decode phase only, excludes the first token	ms/token
Request Latency	e2e_latency	$e2e = TTFT + Generation\_time$ — from the first request to the last response	ms
Output Token Throughput	OTT	$\frac{total\_output\_tokens}{T_y - T_x}$ where $T_x$ = first request, $T_y$ = last token received	tok/s
Request Throughput	RPS	$\frac{total\_completed\_requests}{T_y - T_x}$	req/s
Input Sequence Length	ISL	Mean length in tokens of the input prompt	tokens
Output Sequence Length	OSL	Mean length in tokens of the generated response	tokens

Methodological note on ITL versus LLMPerf: GenAI-Perf excludes TTFT from the ITL calculation (pure decode only). LLMPerf includes TTFT in its ITL mean. The numbers are not directly comparable.

Methodological note on TPS versus LLMPerf: GenAI-Perf measures throughput between the first request and the last token received ($T_y - T_x$). LLMPerf uses the total duration of the benchmark including prompt generation and response storage; in a concurrency-1 scenario, that difference can amount to up to 33% in the reported throughput figure.

Console output table (real example, TRT-LLM backend)

The table GenAI-Perf prints for a profile with ISL=200, OSL=100, concurrency=1:

 NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Statistic ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Time to first token (ms) │ 13.68 │ 11.07 │ 21.50 │ 18.81 │ 14.29 │ 13.97 │
│ Inter token latency (ms) │ 1.86 │ 1.28 │ 2.11 │ 2.11 │ 2.01 │ 1.95 │
│ Request latency (ms) │ 203.70 │ 180.33 │ 228.30 │ 225.45 │ 216.48 │ 211.72 │
│ Output sequence length │ 103.46 │ 95.00 │ 134.00 │ 122.96 │ 108.00 │ 104.75 │
│ Input sequence length │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │
│ Output token throughput (per sec) │ 504.02 │ N/A │ N/A │ N/A │ N/A │ N/A │
│ Request throughput (per sec) │ 4.87 │ N/A │ N/A │ N/A │ N/A │ N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

(Source: docs.nvidia.com/deeplearning/triton-inference-server — LLM tutorial)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

The profile subcommand measures a fixed operating point (concurrency or request-rate, synthetic ISL/OSL).

Example 1: vLLM endpoint (chat, streaming)

# 1. Bring up the vLLM server
docker run -it --net=host --rm --gpus=all \
 vllm/vllm-openai:latest \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --dtype float16

# 2. Profile with GenAI-Perf
genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --concurrency 16 \
 --synthetic-input-tokens-mean 1024 \
 --synthetic-input-tokens-stddev 128 \
 --output-tokens-mean 256 \
 --output-tokens-stddev 0 \
 --output-tokens-mean-deterministic \
 --request-count 200 \
 --warmup-request-count 20 \
 --tokenizer meta-llama/Llama-3.1-70B-Instruct

Example 2: completions endpoint (no streaming)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type completions \
 --concurrency 32 \
 --synthetic-input-tokens-mean 512 \
 --synthetic-input-tokens-stddev 64 \
 --output-tokens-mean 128 \
 --request-count 200 \
 --warmup-request-count 20

Example 3: NVIDIA NIM (TRT-LLM backend endpoint, via Triton)

genai-perf profile \
 -m meta/llama-3.1-70b-instruct \
 --backend tensorrtllm \
 --streaming \
 --concurrency 8 \
 --synthetic-input-tokens-mean 2048 \
 --synthetic-input-tokens-stddev 0 \
 --output-tokens-mean 512 \
 --output-tokens-mean-deterministic \
 --request-count 100 \
 --warmup-request-count 10 \
 --generate-plots

Example 4: request-rate instead of concurrency (Poisson)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --request-rate 5.0 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 300 \
 --warmup-request-count 30

Main flags and what they do

Flag	Typical values	Function
`--endpoint-type`	`chat`, `completions`, `embeddings`	Type of OpenAI-compatible endpoint
`--streaming`	(boolean)	Enables SSE streaming; required to measure real TTFT and ITL
`--concurrency`	1, 8, 16, 32, 64	Number of concurrent requests maintained; GenAI-Perf guarantees N active at all times
`--request-rate`	1.0, 5.0, 10.0	Constant arrival rate (req/s); does not guarantee N active
`--synthetic-input-tokens-mean`	128–8192	Mean of synthetic ISL tokens
`--synthetic-input-tokens-stddev`	0–512	Standard deviation of the ISL (0 = fixed)
`--output-tokens-mean`	64–2048	Mean of target OSL tokens
`--output-tokens-mean-deterministic`	(boolean)	Sets the minimum output tokens = target mean (more precise with TRT-LLM)
`--request-count`	100–2000	Number of requests to benchmark
`--warmup-request-count`	10–50	Warm-up requests discarded from the metrics
`--generate-plots`	(boolean)	Generates PNG plots of TTFT vs ISL, ITL vs token position, etc.
`--tokenizer`	HF model id	Tokenizer for counting tokens; mandatory when ISL/OSL matter
`--backend`	`tensorrtllm`, `vllm`	Triton backend (for serving directly via Triton without an OpenAI endpoint)
`--extra-inputs`	`ignore_eos:true`	Extra request parameters; `ignore_eos:true` guarantees a consistent OSL

Note on --concurrency versus --request-rate: NVIDIA recommends using --concurrency. With --request-rate, if the rate exceeds the engine’s throughput, the queue grows without bound and the metrics stop being stable. With --concurrency, there are always exactly N active requests, which gives a clean latency-throughput curve.

Output artefacts

GenAI-Perf dumps all results into an artifacts/ directory:

Artefact	Format	Content
`profile_export_genai_perf.json`	JSON	Full metrics (avg, min, max, P75, P90, P99) + CLI arguments used
`profile_export_genai_perf.csv`	CSV	Exported console tables, ready to import into Excel/pandas
`profile_export.json`	JSON	Raw perf_analyzer data (per-request traces)
`inputs.json`	JSON	Synthetic payloads sent (for reproducibility)
PNG plots	PNG	TTFT analysis, Request latency, TTFT vs ISL, ITL vs position, ISL vs OSL (with `--generate-plots`)

The `analyze` subcommand: automatic concurrency sweep

The analyze subcommand sweeps multiple values of a parameter in a single command and generates a summary CSV.

Concurrency sweep (the most used)

# Sweeps concurrencies 1, 2, 4, 8, 16, 32, 64, 128, 256
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type concurrency \
 --sweep-range 1:256 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Request-rate sweep

# Sweeps 2, 4, 6, 8, 10, 12 req/s
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type request_rate \
 --sweep-list 2,4,6,8,10,12 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

ISL sweep (to study the effect of prefill)

genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type input_sequence_length \
 --sweep-list 256,512,1024,2048,4096 \
 --concurrency 16 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Artefact structure of `analyze`

For a sweep with --sweep-type concurrency --sweep-range 1:32:

artifacts/
llama70b-openai-chat-concurrency1/
inputs.json
profile_export.json
profile_export_genai_perf.json
profile_export_genai_perf.csv
llama70b-openai-chat-concurrency2/
...
llama70b-openai-chat-concurrency4/
...
[...]
analyze_export_genai_perf.csv ← summary of every scenario
checkpoint.json ← allows interrupted sweeps to be resumed

Summary CSV format

Config Name,Concurrency,ISL,p99 TTFT (ms),p99 ITL (ms),p99 Request Latency (ms),Avg. OTT (tok/s),RPS
llama70b_run_0,1,1024,45.2,6.8,1823.4,128.3,0.55
llama70b_run_1,4,1024,48.1,7.1,1901.2,510.7,2.18
llama70b_run_2,8,1024,61.3,7.9,2142.5,989.2,4.12
llama70b_run_3,16,1024,98.7,9.4,2891.3,1741.6,6.88
llama70b_run_4,32,1024,284.1,14.2,4320.8,2134.2,7.14
llama70b_run_5,64,1024,892.4,28.7,8741.2,2251.8,7.23

The summary CSV also includes a second table with GPU metrics (P99 power, energy, utilisation, memory) when they are captured via Triton’s telemetry URLs (--server-metrics-urls http://localhost:8002/metrics).

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

The choice of tool changes the number. They are not comparable with each other without adjustment.

Dimension	GenAI-Perf	GuideLLM	LLMPerf	vLLM bench serve
Origin / maintainer	NVIDIA / Triton team (retired Apr 2026)	Red Hat / vLLM project	Anyscale / Ray	vLLM project
Class	multi-process load generator	multi-process load generator	multi-process load generator	single-process micro-bench
Supported endpoints	OpenAI-compatible + KServe + native Triton	OpenAI-compatible	OpenAI-compatible	native vLLM
Main load mode	fixed concurrency N (recommended) or constant request-rate	synchronous, concurrent, poisson, throughput, automatic sweep	batches of N concurrent (with a draining period at the end)	request-rate, concurrency
Draining period	NO — guarantees N active at all times	NO (poisson)	YES — at the end of each batch the system empties, concurrency drops to 0	N/A
Automatic sweep	`analyze` (concurrency, request-rate, ISL, OSL)	`--rate-type sweep` (idle to saturation, 10 rounds)	manual (several runs)	`vllm bench sweep serve`
LLM metrics	TTFT, ITL, e2e latency, OTT, RPS, ISL/OSL	TTFT, ITL/TPOT, throughput, goodput under SLO	TTFT, ITL (includes TTFT in the mean), TPS (total benchmark duration)	TTFT, TPOT, throughput
ITL difference	excludes TTFT: $(e2e - TTFT) / (N_{tok}-1)$	excludes TTFT (same as GenAI-Perf)	includes TTFT in the mean	excludes TTFT
TPS difference	$total\_tokens / (T_y - T_x)$	throughput + goodput under SLO	$total\_tokens / (T_{end} - T_{start})$ — up to 33% lower	$total\_tokens / (T_y - T_x)$
Goodput / SLO	NO	YES (the key differentiator)	NO	NO
Exports	JSON + CSV + PNG + checkpoint	JSON + YAML + CSV + interactive HTML	JSON	console / JSON
Built-in warm-up	YES (`--warmup-request-count`)	YES (warm-up requests)	NOT native	YES
Real datasets	OpenOrca, CNN DailyMail, JSONL file, moon_cake	file with a traffic distribution	JSONL file	sharegpt, random
GPU telemetry	YES (via Triton’s server-metrics-urls → summary CSV)	NOT native (integrate DCGM separately)	NO	NO
When to choose it	fine profiling of one operating point, comparing configs with the same harness, integration with NIM/Triton	SLO-driven sweep, finding the knee, sizing replicas	quick endpoint validation in the Ray ecosystem	iterating on vLLM flags

Fair-comparison rule: never cross numbers from different tools for the same conclusion. If vLLM and TRT-LLM are being compared, use GenAI-Perf against both, with the same ISL/OSL, the same concurrency and the same --warmup-request-count. The tool is the constant; the engine is the variable.

Profiling on 4×H100 SXM 80 GB: a full sweep example

Generic reference hardware: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP16, ISL=1024 tokens, OSL=256 tokens, vLLM endpoint.

Illustrative result of a concurrency sweep (1→64)

Concurrency	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	ITL P99 (ms)	OTT (tok/s)	RPS
1	38	45	6.4	7.1	130	0.51
4	40	51	6.8	8.2	510	2.00
8	46	62	7.3	9.4	980	3.83
16	68	98	8.9	12.1	1,720	6.72
32	145	284	13.7	21.4	2,130	8.32
48	380	740	22.3	38.6	2,280	8.90
64	890	1,820	41.2	78.3	2,340	9.14

How to read it: throughput saturates around concurrency 48–64, while TTFT P99 already exceeds 500 ms from concurrency 32 onwards. The knee (the last point with TTFT P99 < 500 ms) is at concurrency 16 with 1,720 useful tok/s. Anyone reporting “2,340 tok/s” (concurrency 64) is describing the maximum throughput, at a point where the system meets no interactive chat SLO at all.

Formulas for the key metrics

$$ \text{ITL} = \frac{e2e\_latency - TTFT}{N_{output\_tokens} - 1} $$ $$ \text{OTT} = \frac{\sum output\_tokens}{T_y - T_x} $$ $$ \text{RPS} = \frac{N_{requests}}{T_y - T_x} $$

where $T_x$ is the timestamp of the first request sent and $T_y$ the timestamp of the last token received from the last request.

Synthetic ISL/OSL profiling: what to configure for each use case

The ISL/OSL parameters determine which engine of the system is stressed: a long ISL stresses prefill (KV-cache, TTFT); a long OSL stresses decode (ITL, memory bandwidth).

Use case	Typical ISL	Typical OSL	`--synthetic-input-tokens-mean`	`--output-tokens-mean`
Short interactive chat	~300 tok	~100 tok	300	100
Code copilot	~800 tok	~200 tok	800	200
Document summarisation	~2000 tok	~256 tok	2000	256
RAG with long context	~4096 tok	~512 tok	4096	512
Batch without latency	~512 tok	~1024 tok	512	1024

The --extra-inputs ignore_eos:true option disables the engine’s EOS token, forcing exactly the requested tokens to be generated. Essential for measuring a consistent OSL on TRT-LLM; optional on vLLM.

Honest methodology: how not to measure badly

1. Warm-up is mandatory

Without warm-up, the first requests include the KV-cache loading latency, the engine’s autotuning and the initialisations. GenAI-Perf applies a sliding window to detect stability (max/min ratio within a margin over the last 3 measurements), but the explicit --warmup-request-count discards the first N requests from the metrics. Recommended minimum value: 10 requests (or 5% of the total if the total is large).

2. The concurrency knee: extend the sweep until you see it

Throughput saturates once the engine’s effective max batch size is reached. If the sweep stops before saturation is reached, the “maximum capacity” reported is the last concurrency tested, not the real knee. The rule: the sweep must reach a point where TTFT P99 has doubled relative to the low-load value (or where OTT stops growing by more than 5% between steps). Only then is the knee visible.

For a sweep with --sweep-range 1:256, GenAI-Perf sweeps 1, 2, 4, 8, 16, 32, 64, 128, 256 (powers of 2). On most 70B models over 4×H100, saturation occurs between concurrency 32 and 64; the range 1:128 covers the knee with room to spare.

3. Concurrency versus request-rate: choosing correctly

Mode	Behaviour	When to use
`--concurrency N`	Always N active requests; when one finishes, the next is sent	Clean latency-throughput curve; comparison between configs
`--request-rate R`	1 request is sent every 1/R seconds; the queue can grow without bound	Simulating traffic with arrivals at a constant rate

NVIDIA recommends --concurrency for benchmarking. With --request-rate, if R exceeds the engine’s maximum throughput, the queue grows indefinitely and the metrics become unstable.

4. Single-process versus multi-process bias

GenAI-Perf is multi-process: it maintains N concurrent requests using perf_analyzer’s internal manager. There is no risk of client saturation. This sets it apart from vllm bench serve (single-process), which can saturate on the client at high concurrency and give inflated throughput metrics.

5. `ignore_eos` and an inconsistent OSL

Without --extra-inputs ignore_eos:true, the engine may generate fewer tokens than requested (if the model produces an EOS earlier). This makes the real OSL vary between requests and the ITL become inconsistent between runs. For reproducible benchmarks with a controlled OSL: always enable ignore_eos:true.

6. The correct tokenizer

The --tokenizer flag must point to the model being served. If it is omitted, GenAI-Perf uses a default tokenizer that can give different ISL/OSL in the real model’s tokens, making the throughput numbers in tok/s not comparable between models with different vocabularies.

7. One run = a data point; three or more = a number

Variance between runs under identical conditions can be 5–10% on tail latency metrics (P99). A single profile is not a defensible number; the mean of three runs with the same config is.

Connection with the cost and energy track

GenAI-Perf can capture GPU metrics in the same summary CSV if --server-metrics-urls http://localhost:8002/metrics is passed (Triton’s metrics endpoint, which exposes power and utilisation via DCGM). The analyze summary CSV then includes a second table with P99 power (W), energy (MJ), utilisation (%) and memory (GB) per GPU and per scenario.

With that data, a single sweep yields the three axes of the scorecard:

Axis	Source	Metric
Performance	GenAI-Perf	OTT (tok/s), TTFT P99, ITL P99
Energy	DCGM via server-metrics-urls	Power (W) → $J/token = W / OTT$
Cost	node price	EUR/hour → $EUR/token = EUR_{hour} / (OTT \times 3600)$

The formula for cost per million tokens:

$$ CPM = \frac{EUR_{hour}}{OTT \times 3{.}6 \times 10^3} $$

where $OTT$ is the output token throughput in tok/s and $CPM$ is the cost per million tokens in EUR.

It connects with GPU observability with DCGM for the power capture, and with anatomy of an LLM request for the breakdown of phases (prefill versus decode) that TTFT and ITL quantify.

Status 2026: GenAI-Perf → AIPerf

NVIDIA announced in April 2026 that GenAI-Perf moves to passive maintenance mode (no new features) and that the active successor is AIPerf (github.com/ai-dynamo/aiperf). AIPerf adds automatic detection of the saturation point (estimatedCapacity) and is integrated into the NVIDIA Dynamo pipeline.

For new benchmarks in production: migrate to AIPerf. To reproduce historical runs documented with GenAI-Perf, or to use NVIDIA’s NIM benchmarking artefacts (which use GenAI-Perf): stay on GenAI-Perf 25.x.

The article LLM benchmark tools, one by one covers the full comparison including AIPerf. The article GuideLLM: SLO validation under load details the SLO-driven sweep that GenAI-Perf does not do natively. For the general methodology context: LLM benchmarking frameworks, state of the art.

Sources

NVIDIA · GenAI-Perf — README oficial — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · Tutorial LLM con GenAI-Perf (comandos y tablas de salida reales) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
NVIDIA · GenAI-Perf Analyze subcommand (sweep, CSV resumen, checkpoint) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/analyze.html
NVIDIA · NIM LLM Benchmarking — Métricas (definiciones TTFT, ITL, TPS, RPS, fórmulas exactas) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
NVIDIA · NIM LLM Benchmarking — Parámetros y buenas prácticas (ISL/OSL, concurrencia vs request-rate, ignore_eos) — https://docs.nvidia.com/nim/benchmarking/llm/latest/parameters.html
NVIDIA Technical Blog · Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API — https://developer.nvidia.com/blog/measuring-generative-ai-model-performance-using-nvidia-genai-perf-and-an-openai-compatible-api/
GitHub · triton-inference-server/perf_analyzer (genai-perf) — https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
GitHub · ray-project/llmperf (métricas y diferencias metodológicas) — https://github.com/ray-project/llmperf
PyPI · nvidia-genai-perf-eval — https://pypi.org/project/nvidia-genai-perf-eval/
Macnica · Benchmarking LLM Applications Part 1: What is GenAI-Perf? — https://www.macnica.co.jp/en/business/semiconductor/articles/nvidia/145977/

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

Tue, 02 Jun 2026 04:30:00 +0200

This post complements those on Capacity planning for on-premise LLM inference (which assumed a pure NVIDIA cluster), Seven layers of the stack (which also did not get into hardware heterogeneity) and The LLM inference router (where capability routing makes full sense once there is mixed hardware). It is the missing piece for talking about “hardware sovereignty” without reducing it to “which vendor do I pick”.

TL;DR

A production LLM inference cluster in 2026 can stop being monolithically NVIDIA if it accepts heterogeneity as an architectural decision. The motivation is not theory but three measurable operational advantages. (1) Cost: an Intel Xeon 6 with AMX (Advanced Matrix Extensions) delivers 7B INT4 at ~80 tok/s while serving embeddings and reranking at a fraction of the cost of dedicating an H100 to that task; the capacity planning adds up better with an Intel CPU handling the cheap work and an NVIDIA H100 the big LLM. (2) Sovereignty and supply chain diversification: NVIDIA holds roughly 94 % of the AI accelerator market (November 2025), single-vendor dependency with all its risks; Intel manufactures in Europe (Leixlip operational, Magdeburg planned) against a design-only NVIDIA with TSMC as foundry, which for a Spanish or European organisation subject to ENS / NIS2 / EU AI Act is a real hedging argument. (3) Edge: an Intel NUC with a Lunar Lake CPU (48 TOPS NPU) or Panther Lake (50 TOPS NPU + 120 TOPS Xe3 = 180 TOPS platform) runs 7B INT4 models at usable speed, which opens up the “branch office with local inference + central DC for complex cases” pattern. Relevant Intel hardware in June 2026: Intel Gaudi 3 (128 GB HBM2e, 1,835 TFLOPS BF16/FP8, 3.67 TB/s; a direct competitor to the H100, Intel claims +20 % on Llama 2 70B but Signal65 published H200 9× over Gaudi 3 on Llama 3.1 405B, you have to cite both; Falcon Shores cancelled in January 2025, Jaguar Shores 2026 as the restart bet, Gaudi 4 confirmed as never happening); Intel Xeon 6 with AMX (up to 288 E-cores on Sierra Forest or 86 P-cores on Granite Rapids, 1,024 FLOPS BF16/cycle/core with AMX, Intel claims 2.7× tok/s vs EPYC 9965 on the vLLM CPU backend); Intel Arc Pro B60 (Battlemage, 24 GB GDDR6, 456 GB/s, 197 TOPS INT8, launched September 2025, with a dual-GPU 48 GB variant and a “Battlematrix” rack with 8× = 192 GB VRAM); Intel NUC with NPU (Lunar Lake 48 TOPS, Arrow Lake similar, Panther Lake 50 TOPS at CES 2026; realistic for 7-13B INT4, not for the 30-70B Intel claims in its marketing). Software: OpenVINO 2025.3 with the GenAI API and vLLM-OpenVINO; IPEX-LLM with integrations into llama.cpp, vLLM, HF, LangChain; vLLM CPU backend with AMX; llama.cpp SYCL (better than Vulkan on Arc). Four canonical patterns: embeddings + reranker on Intel alongside the LLM on NVIDIA; guardrails + PII redaction on a near-edge NUC; speculative drafter on a NUC near the user and target on the H100; NUC dev workstations. Unified observability via DCGM + habana-metric-exporter + intel-gpu-exporter + Intel PCM federated in Prometheus. Pitfalls: tokenizer mismatch between engines, edge↔central round-trip latency, FP8 Hopper ≠ INT8 AMX in quality, version synchronisation. Applied to a generic cluster: central DC 4×H100 SXM + Xeon 6 AMX sidecar + 6-12 Intel NUCs in branch offices. Critical disclaimer: as of June 2026 there are no publicly verifiable cases of a mixed NVIDIA + Intel deployment in European banking or government; the pattern is emergent architecture and worth recommending, not established practice with an industrial track record.

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

A serious factory has several machines with different purposes, not one universal machine. A 200-tonne hydraulic press for heavy stamping; a bench lathe for turned parts; a 3D printer for quick prototypes; a pick-and-place robot for SMD. Each machine does what it does better than the others in its niche, and the plant manager sizes the mix according to the real product portfolio, not according to fashion. Buying three hydraulic presses because “they are the most impressive” when 60 % of the work is turned parts wastes capital: the lathe is cheaper, faster in its niche and frees the press for what genuinely needs it.

An LLM inference cluster with an NVIDIA H100 dedicated to computing embeddings for a RAG corpus is using a hydraulic press to drill bolts. The H100 is magnificent for a 70B LLM in BF16 with concurrency of 40+; for embeddings of an 800-token document with bge-m3, what you need is an Intel Xeon 6 with AMX at a fraction of the cost and power draw. A cluster that wants to serve light guardrails (Llama Guard 4 8B) on every request, with a 50 ms budget, does not need that guardrail on an H100 either: an Intel NUC with a 48 TOPS NPU covers the case with room to spare.

The heterogeneous factory is not theoretical elegance: it is maximising the useful utilisation of fixed capital. Nor is the heterogeneous LLM inference cluster.

Three operational reasons for heterogeneity

Reason 1 — cost

An H100 SXM 80 GB running 24/7 draws ~700 W (real measurement at the wall ~697 W with vLLM Llama 3.1 405B batch=4) and represents between 25,000 € and 35,000 € of amortised hardware. An Intel Xeon 6 with AMX (Granite Rapids 86 cores or Sierra Forest 288 E-cores) draws 350-500 W for the socket and costs a fraction. The operational split: the H100 is reserved for the big LLM (Llama 70B BF16 or FP8, where its HBM3 and FP8 tensor cores are worth their weight); the Xeon AMX absorbs embeddings (bge-m3, e5-large), reranking (bge-reranker-v2-m3), small models (Llama 3.2 1B / 3B INT4) and offline batch processing. It is the same logic as the capacity planning taken one step further: instead of budgeting KV cache VRAM only on the H100, budget each workload on the silicon where its arithmetic intensity fits best.

Reason 2 — sovereignty and supply chain diversification

As of November 2025, NVIDIA holds approximately 94 % of the AI accelerator market. That concentration is risk. For an organisation subject to ENS / NIS2 / EU AI Act, depending on a single supplier whose foundry is concentrated in Taiwan (TSMC) introduces supply chain vulnerabilities that recent regulations (NIS2, supply chain provisions) are starting to require you to document and mitigate. Intel diversifies: it has its own fabs in Europe (Leixlip operational in Ireland; Magdeburg planned in Germany, with EU Chips Act funding), which for a Spanish or European public sector client is a real contractual argument, not marketing.

Mandatory disclaimer: the Intel roadmap after Falcon Shores is unstable. Intel cancelled Falcon Shores in January 2025 and relegated Gaudi 4 to “will never exist”; the restart bet is Jaguar Shores in 2026 as a rack-scale platform, still with no confirmed public specs. Diversification is strategically correct, but assuming continuity of the Intel roadmap at NVIDIA’s level in 2026 would be naive. The operational strategy: Intel for workloads where lock-in is lower (CPU for embeddings, NUC for light edge, replaceable by AMD/Apple/SiFive if Intel pivots again), NVIDIA for the big LLM where the maturity of the software stack still has no rival.

Reason 3 — edge

The “everything travels to the central DC” pattern breaks in three cases: latency (branch office 100+ ms away from the DC, unacceptable for chat), data sovereignty (prompts with personal or classified data that must not leave the local perimeter), and offline operation (branch office with intermittent connectivity). The Intel NUC with a modern CPU (Lunar Lake / Arrow Lake / Panther Lake) brings 48-50 TOPS NPU + 100-180 TOPS Xe2/Xe3 iGPU in a box of 0.5-1.5 L volume drawing 30-65 W. 7B INT4 models run at usable speed; with more aggressive quantisation (Q3_K) a Llama 13B fits. For branch offices with RAG over a local corpus + a 7B LLM + guardrails, the NUC is perfect.

Relevant Intel hardware (June 2026)

Part	Memory	Key performance	Launch	Status
Intel Gaudi 3	128 GB HBM2e, 3.67 TB/s	1,835 TFLOPS BF16/FP8; 1,200 GB/s networking	Apr 2024	Active; successor Jaguar Shores 2026 (not Gaudi 4)
Intel Xeon 6 (Granite Rapids)	DDR5 + MRDIMM	86 P-cores, AMX 1,024 FLOPS BF16/cycle/core	2024-2025	Active
Intel Xeon 6 (Sierra Forest)	DDR5	288 E-cores	2024	Active
Intel Arc Pro B60 (Battlemage)	24 GB GDDR6, 456 GB/s	197 TOPS INT8; 12.28 TFLOPS FP32	Sep 2025	Active; dual 48 GB variant, rack 8× = 192 GB
Intel Data Center GPU Max	128 GB HBM	successor to Ponte Vecchio	discontinued	Discontinued Jan 2026
Intel NUC (Lunar Lake)	DDR5x	NPU 48 TOPS + Xe2 67 TOPS = 120 TOPS platform	2024	Active
Intel NUC (Arrow Lake)	DDR5	NPU 13 TOPS + Xe iGPU	2024	Active (less NPU than Lunar)
Intel NUC (Panther Lake)	DDR5x	NPU 50 TOPS + Xe3 120 TOPS = 180 TOPS platform	CES Jan 2026	Rolling out

Intel Gaudi 3 — the critical note on the marketing

Intel publishes that Gaudi 3 delivers +20 % throughput vs H100 on Llama 2 70B inference and 2× price/performance. The figure appears in the official whitepaper and in launch presentations. However, Signal65 (an independent firm) published in 2025 that the H200 beats Gaudi 3 by a factor of 9× on Llama 3.1 405B. The discrepancy matters: both numbers can be true for their specific benchmarks (Llama 2 70B FP16 vs Llama 3.1 405B FP8) but the operational conclusion changes radically depending on which one you keep.

This post’s recommendation: treat Gaudi 3 as a valid option for Llama-class 70B in BF16/FP8 where Intel claims parity or advantage, not for frontier 200B+ models where NVIDIA keeps a clear margin. And consider the roadmap risk: Gaudi 4 will never exist; the line’s successor is Jaguar Shores 2026 with a completely new rack-scale architecture, a discontinuity rather than an evolution.

Intel Xeon 6 with AMX — the CPU workhorse

The Advanced Matrix Extensions (AMX) are the non-obvious piece. Each Granite Rapids P-core executes up to 1,024 BF16 FLOPS per cycle via AMX, which turns a Xeon 6 with 64-86 cores into a respectable matrix accelerator for small and mid-sized models. Real reported figures: Llama 3.2 INT4 at ~57 tok/s with AMX vs 28 tok/s without AMX (a clean 2× factor). For serving 7B INT4 with the vLLM CPU backend + AMX, Intel claims 2.7× tok/s vs EPYC 9965, a figure with Intel bias but qualitatively corroborated by LMSYS in their DeepSeek R1 671B deployment on Xeon 6 + SGLang.

Operational use case: embeddings and reranking on a Xeon 6 sidecar next to the H100 cluster. Models such as bge-m3 (multilingual embedding) or bge-reranker-v2-m3 run at acceptable throughput on CPU AMX; they do not deserve a dedicated H100. Freeing the H100 for the 70B LLM raises the cluster’s effective RPS without buying more GPUs.

Intel Arc Pro B60 and Battlematrix

Launched in September 2025, the Arc Pro B60 (Battlemage) brings 24 GB GDDR6 with 456 GB/s of bandwidth and 197 TOPS INT8 at 200 W. Maxsun has a dual-GPU 48 GB variant. The “Battlematrix” rack configuration combines 8 units = 192 GB of aggregate VRAM. The interesting point: at a cost far below an H100 SXM 80 GB, which makes it a candidate for 30-70B INT4-INT8 LLMs served via OpenVINO or llama.cpp SYCL.

Phoronix verified that on SYCL the Arc Pro B70 reaches parity with the Radeon PRO W7900 (previous AMD generation) on DeepSeek R1 Llama 8B pp512. The Vulkan backend loses heavily (~1/4 of SYCL performance); for Arc Pro, always SYCL.

Intel NUC with NPU — the edge node

Intel NUCs with a Lunar Lake CPU (Core Ultra Series 2) bring NPU 4 with 48 TOPS and a platform total of 120 TOPS adding the Xe2 iGPU and CPU AVX. Panther Lake (CES January 2026) goes up to NPU 5 = 50 TOPS + Xe3 120 TOPS = 180 TOPS platform.

Intel states that Panther Lake “runs 30-70B models locally”. Reality check: it is marketing. A 30-70B INT4 fits in RAM (DDR5x 32-64 GB) but sustained speed with Q4_K_M quantisation on a NUC hovers around 2-8 tok/s; comfortable for occasional use, not for serving traffic. The NUC’s real sweet spot is 7B INT4 at 20-40 tok/s on the iGPU/NPU, perfect for a client branch office with casual queries.

Intel software — the relevant stack

OpenVINO 2025.3 (June 2026) is the central piece. It supports single-command deployment via the OVMS CLI with automatic download from the HF Hub; it integrates OpenVINO GenAI with a C++/Python API for generative pipelines; it exposes a vLLM v1-compatible API (vLLM-OpenVINO). GGUF model support: DeepSeek Distill, Qwen 2/2.5, Llama 3. Optimisations: Sage Attention (first token with long prompts), per-channel KV-cache compression.

Intel Extension for PyTorch (IPEX), XPU version 2.8.10+xpu, adds Intel backends to PyTorch. IPEX-LLM is the subproject that integrates with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM and DeepSpeed. May 2025: it ran DeepSeek V3/R1 671B and Qwen3MoE 235B on 1-2 Arc A770/B580 cards with FlashMoE.

vLLM CPU backend, the CPU branch of vLLM with AMX optimisations. For 7B INT4 on a 4th-gen Xeon with AMX: 12-50 tok/s; with a Xeon Gold 6530 + INT4: ~80 tok/s. Academic figures (arXiv 2410.04466).

llama.cpp SYCL, the recommended backend for Arc; Vulkan works but hovers around 1/4 of SYCL performance on an Arc B580. SYCL reaches parity with the previous AMD generation.

Habana SynapseAI, the Gaudi 3 stack. The PyTorch bridge habana_frameworks.torch registers the hpu device; it integrates with torch.compile. It is not a complete port to oneAPI but a partial integration via oneMKL. That implies the Gaudi ecosystem keeps a certain separation from Intel’s general oneAPI, relevant in view of the hypothetical Jaguar Shores and future unification.

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

The most mature and the easiest to adopt. In a typical RAG system, each user request invokes:

Query embedding (50 ms on an H100, 80 ms on a Xeon AMX, 30 ms on an Arc Pro B60).
Vector search (Qdrant / Milvus / Chroma; latency ~10-30 ms).
Reranker over the top-k candidates (60 ms on an H100, 100-150 ms on a Xeon AMX).
LLM over the augmented prompt (200-500 ms TTFT, 30-50 ms/token).

Steps 1 and 3 are memory-bound and relatively small (100M-1B models): a Xeon 6 with AMX (the Arc Pro B60 is faster but is already a dedicated GPU) does the job at a hardware cost that is a fraction of a dedicated H100. Step 4 stays on NVIDIA because that is where its tensor architecture + HBM3 + FP8 is worth what it costs.

Operational implication: a Xeon 6 sidecar (~40 cores, ~10-15 k€) serving embeddings + reranking frees the equivalent of 1-2 H100s of “cheap” load, recovering that capacity for the big LLM. Clear ROI in sizing.

Pattern 2 — guardrails + PII redaction on a near-edge NUC

A bank branch, a medical practice or a law office generates prompts with personal or classified data. Sending those prompts to the central DC (even if it is corporate on-premise) can clash with local retention policies or with specific compliance (GDPR, professional privilege).

The pattern: the NUC in the branch office runs two critical steps before the round-trip:

PII redaction with Presidio (CPU-only, fast) or Llama Guard 4 8B on the NUC’s NPU + iGPU. It replaces names, tax IDs and account numbers with placeholders.
Light guardrails (PromptGuard 2 86M, Llama Guard 4 8B) on the NPU + iGPU. It filters prompt injection, jailbreaks and prohibited content.

Only afterwards does the redacted prompt travel to the central DC so the big LLM can answer. The response comes back to the NUC, which rehydrates the placeholders with the real values before showing it to the user. The sensitive data never leaves the branch office.

Costs: a Panther Lake NUC is ~1,500-2,500 € per unit, scalable to dozens of branch offices with no additional central GPU cost. Latency: 50-150 ms for the edge step before the DC’s 200-500 ms round-trip.

Pattern 3 — speculative decoding drafter on a NUC

Speculative decoding uses a small drafter that proposes γ tokens and a large target that accepts or rejects them in a single forward pass. If the drafter is geographically close to the user (a NUC in the branch office) and the target is in the central DC, the client’s perceived latency falls further still.

Setup: drafter Llama 3.2 1B INT4 on a NUC + target Llama 3.1 70B FP8 on the central H100. The NUC generates γ=4 tokens in ~50 ms locally; the target verifies them in one pass (40-80 ms including round-trip); if the acceptance rate is > 60 %, effective TTFT falls ~50 % compared with Llama 70B without speculative decoding.

An important constraint: drafter and target must share a tokenizer. Llama 3.2 1B and Llama 3.1 70B have compatible tokenizers. Mixing a Llama drafter with a Qwen target breaks the pattern.

Pattern 4 — NUC dev workstations

Dev / CI iterating on prompts, evals and retrieval logic does not need a production GPU to validate correctness. A NUC with Llama 3.2 3B INT4 runs the end-to-end functional tests (including embeddings + retrieval + LLM + guardrails) at a tenth of the cost of iterating on an H100. Only the last pre-prod smoke test uses the production cluster.

A mature pattern in organisations with many developers and scarce production GPU. Iteration that is 10× faster and cheaper translates into feature delivery speed.

Unified observability in a heterogeneous cluster

The GPU observability post covered DCGM Exporter for NVIDIA. In a mixed cluster you need more:

Hardware part	Exporter	Key metrics
NVIDIA H100/A100	`nvidia/dcgm-exporter`	DCGM_FI_DEV_* + DCGM_FI_PROF_*
Intel Gaudi 3	`HabanaAI/habana-metric-exporter`	habana_hpu_utilization, habana_hbm_used
Intel Arc Pro	`intel/intel-gpu-exporter` (unofficial; alternatives exist)	xe_engine_utilization, xe_memory_used
Intel Xeon CPU + AMX	`prometheus/node-exporter` + Intel PCM	cpu_amx_utilization (via PCM)
Intel NUC (NPU+iGPU)	`intel/intel-gpu-exporter` + custom NPU exporter	npu_utilization, xe_iGPU

All federated into a single Prometheus + Grafana. The dashboards are organised by hardware family (NVIDIA, Intel server, Intel edge) plus an aggregate “heterogeneous cluster” view with per-tenant SLOs combining all four.

Cardinality: ~1.5-2× that of a pure NVIDIA cluster. Manageable with Thanos / Mimir for long retention.

Capability routing — from the L7 router to the heterogeneous one

The LLM inference router stops being a version selector for the same model and becomes a capability dispatcher:

models:
 - name: "llama-70b-chat"
 endpoint: "vllm-llama70b.inference.svc:8000"
 backend: nvidia-h100
 capabilities: [chat, tool_use, json_mode]
 - name: "embedding-multilingual"
 endpoint: "ipex-bge-m3.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [embeddings]
 - name: "reranker-multilingual"
 endpoint: "ipex-bge-reranker.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [reranking]
 - name: "guardrail-prompt-injection"
 endpoint: "openvino-llama-guard.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [guardrails, redact-pii]
 region: sucursal-01
 - name: "llama-3b-draft"
 endpoint: "openvino-llama-3b.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [speculative-drafter]
 region: sucursal-01
 target_model: "llama-70b-chat"

The router resolves model=embedding-multilingual → Intel Xeon; model=llama-70b-chat → H100; model=guardrail-prompt-injection with region=sucursal-01 → local NUC. If the branch office NUC goes down, failover to an equivalent replica in the central DC, accepting the latency cost.

LiteLLM Proxy, NVIDIA Dynamo and Envoy AI Gateway support this capability routing. The non-obvious piece: the router has to know the compatible tokenizer between drafter and target for pattern 3, which is modelled in extra catalogue metadata.

Specific pitfalls

Tokenizer mismatch between engines. OpenVINO with a Llama 3.2 GGUF and vLLM with the same nominal Llama 3.2 can use slightly different tokenizers (chat template, special tokens). Validate token identity with tokenizer.encode("hola") on both sides before assuming interchangeability. For speculative decoding, a single different token breaks the pattern.

Edge ↔ central round-trip latency. Patterns 2 and 3 assume the NUC and the DC are on the same corporate WAN with controlled latency. If the branch office is on 4G/5G with 100-200 ms of jitter, the speculative drafter compensates for nothing; on the contrary, it adds latency. Measure before promising.

FP8 Hopper ≠ INT8 AMX in output quality. The operator assumes a request that runs FP8 on an H100 and INT8 on a Xeon AMX will produce the same output. That is not true: the two quantisations have different degradation profiles. If the system expects idempotence (for example, evals with golden output), validate offline that the Intel version reproduces the expected behaviour within tolerance.

Model version synchronisation between sites. The model in the central DC gets updated, but the branch office NUCs keep the old version of the drafter or the guardrail for weeks. Result: divergent behaviour between branches with no easy diagnosis. Policy: the central model and the edge model advance together or within a documented window; the canary extends to the NUC fleet.

Unstable Intel roadmap. Falcon Shores cancelled, Gaudi 4 will never exist, Jaguar Shores 2026 still with no confirmed public specs. Buying Gaudi 3 today is reasonable if the use case justifies the 18-24 months of amortisation; committing an architecture for 5+ years on an Intel accelerator is a riskier bet than the NVIDIA equivalent, at least until Jaguar Shores materialises with a mature software stack.

Absence of public production deployments. As of June 2026, the confirmed Gaudi 3 deployments are IBM Cloud, Dell AI Factory and a handful of early adopters (Bharti Airtel, Bosch, Naver). There is no publicly verifiable case of a mixed NVIDIA + Intel cluster in European banking or government. This pattern is recommended emergent architecture, not practice with an industrial track record. The first adopter takes on a validation cost that a second adopter avoids.

Applied to a generic on-premise cluster

For an organisation with a generic heterogeneous LLM inference cluster:

Central DC: 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink = 16 H100s. Serves the big LLMs (Llama 70B, Mixtral 8×22B, Qwen 72B) in BF16 or FP8.
Xeon 6 sidecar: 2-4 Xeon 6 servers (Granite Rapids 64-86 cores) with AMX, 512 GB DDR5, in the same rack as the H100 cluster. Serves embeddings (bge-m3), reranking (bge-reranker-v2-m3) and small models (Llama 3.2 1B/3B) on the vLLM CPU backend with AMX.
Arc Pro sidecar (optional): 1-2 servers with 4-8× Arc Pro B60 24 GB each (Battlematrix), for 13-30B INT8 models via OpenVINO. Useful if the cost per mid-sized LLM has to come down from the H100.
Edge NUCs in branch offices: 1-2 Panther Lake NUCs per branch, with a 50 TOPS NPU + Xe3 120 TOPS, serving Llama Guard 4 + Presidio + a Llama 3.2 1B INT4 drafter via OpenVINO. Corporate WAN connectivity with latency < 80 ms to the DC.

Estimated envelope: central cluster ~120 kW of GPU peak + ~10-15 kW of Intel sidecars. Edge: ~50 W per NUC, negligible compared with office costs.

Observability: federated Prometheus in the DC + pull scrape from the NUCs (via corporate VPN). Dashboards “NVIDIA GPU fleet”, “Intel server fleet”, “Intel edge fleet” plus a “consolidated SLO” view.

Router: LiteLLM Proxy or NVIDIA Dynamo in the DC, with an extended model catalogue to include Intel backends and regions (sucursal-01, sucursal-02, …). Documented edge→central failover.

What we have not covered (upcoming posts)

Reproducible benchmarks of Llama 70B on Gaudi 3 vs H100 SXM on equivalent hardware, the missing material for making decisions with your own data rather than Intel’s or Signal65’s.
AMD ROCm in the mix: how MI300X / MI355X fit into this heterogeneous pattern and what changes in the router catalogue.
Apple Silicon as edge: M3/M4 Max with a Neural Engine of ~38 TOPS + a 40-core GPU, hardware equivalent to the Panther Lake NUC but with a different software stack (MLX).
Energy cost optimisation: how nvidia-smi -pl 500W + an Intel TDP cap on Xeon 6 cuts the bill by 25-30 % with a 15-20 % throughput loss.
Model CI/CD for an edge fleet: how a rolling update of a Llama Guard reaches 50 branch office NUCs without any of them losing service.

References

Intel Gaudi 3

Intel — Gaudi 3 AI Accelerator White Paper. https://cdrdv2-public.intel.com/817486/gaudi-3-ai-accelerator-white-paper.pdf
Intel — Hot Chips 2024 Gaudi 3 deep dive. https://hc2024.hotchips.org/assets/program/conference/day1/60_HC2024.Intel.RomanKaplan.Gaudi3-0826.pdf
Signal65 / DataCenterDynamics — NVIDIA H200 outperforms Intel Gaudi 3 by factor of 9× across first Llama 3.1 405B benchmark test. https://www.datacenterdynamics.com/en/news/nvidia-h200-outperforms-intel-gaudi-3-by-factor-of-nine-across-first-llama-31-405b-benchmark-test-exclusive/
IEEE Spectrum — Intel Gaudi 3 review. https://spectrum.ieee.org/intel-gaudi-3
Tom’s Hardware — Intel cancels Falcon Shores GPU; Jaguar Shores to be successor. https://www.tomshardware.com/tech-industry/artificial-intelligence/intel-cancels-falcon-shores-gpu-for-ai-workloads-jaguar-shores-to-be-successor

Intel Xeon 6 + AMX

Intel — Xeon 6 (Granite Rapids) Product Brief. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-02/xeon-6-granite-rapids-product-brief.pdf
OpenMetal — Intel AMX AI Inference Performance. https://openmetal.io/resources/blog/intel-amx-ai-inference-performance/
LMSYS — Intel Xeon 6 + SGLang for DeepSeek R1 671B. https://www.lmsys.org/blog/2025-07-14-intel-xeon-optimization/
arXiv 2410.04466 — CPU-LLM benchmarks with AMX.
Intel community blog — Accelerating vLLM Inference on Intel Xeon 6 Processor.

Intel Arc Pro Battlemage

Intel — Arc Pro B60 Graphics Specifications. https://www.intel.com/content/www/us/en/products/sku/243916/intel-arc-pro-b60-graphics/specifications.html
StorageReview — Intel Arc Pro B60 Battlematrix Preview: 192GB VRAM for On-Premise AI. https://www.storagereview.com/review/intel-arc-pro-b60-battlematrix-preview-192gb-of-vram-for-on-premise-ai
Phoronix — Intel Arc Pro B-series review. https://www.phoronix.com/review/intel-arc-pro-b-series

Intel NUC / NPU

HotHardware — Intel CES 2026 Panther Lake is a Go. https://hothardware.com/news/intel-ces-2026-panther-lake-is-a-go
TechPowerUp — Intel Panther Lake Technical Deep Dive.
arXiv 2412.11053 — NITRO: LLM inference on laptop NPU.
Intel — AI PC brings larger LLM development to your desk.

Software

OpenVINO — Release Notes 2025.3. https://www.intel.com/content/www/us/en/developer/articles/release-notes/openvino/2025-3.html
HuggingFace — Deploy with OpenVINO. https://huggingface.co/blog/deploy-with-openvino
Intel — Intel Extension for PyTorch XPU 2.8.10. https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/releases.html
IPEX-LLM — github.com/intel/ipex-llm.
Habana — SynapseAI PyTorch Theory of Operations. https://docs.habana.ai/en/latest/PyTorch/PyTorch_Gaudi_Theory_of_Operations.html

Market context

MLCommons — MLPerf Inference v6.0 benchmark results. https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
Intel newsroom — Gaudi 3 Expanded Availability. https://newsroom.intel.com/artificial-intelligence/intel-gaudi-3-expands-availability-drive-ai-innovation-scale

Sources: the full URLs are linked inline on each reference.

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Nvidia on lo0 — Tech Blog

GenAI-Perf in depth: LLM inference profiling with real data

TL;DR

What GenAI-Perf is and where it fits

Metrics GenAI-Perf produces

Metrics glossary

Console output table (real example, TRT-LLM backend)

Invocation against OpenAI-compatible endpoints

profile mode: one load point

Main flags and what they do

Output artefacts

The analyze subcommand: automatic concurrency sweep

Concurrency sweep (the most used)

Request-rate sweep

ISL sweep (to study the effect of prefill)

Artefact structure of analyze

Summary CSV format

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

Profiling on 4×H100 SXM 80 GB: a full sweep example

Illustrative result of a concurrency sweep (1→64)

Formulas for the key metrics

Synthetic ISL/OSL profiling: what to configure for each use case

Honest methodology: how not to measure badly

1. Warm-up is mandatory

2. The concurrency knee: extend the sweep until you see it

3. Concurrency versus request-rate: choosing correctly

4. Single-process versus multi-process bias

5. ignore_eos and an inconsistent OSL

6. The correct tokenizer

7. One run = a data point; three or more = a number

Connection with the cost and energy track

Status 2026: GenAI-Perf → AIPerf

See also

Sources

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

TL;DR

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

Three operational reasons for heterogeneity

Reason 1 — cost

Reason 2 — sovereignty and supply chain diversification

Reason 3 — edge

Relevant Intel hardware (June 2026)

Intel Gaudi 3 — the critical note on the marketing

Intel Xeon 6 with AMX — the CPU workhorse

Intel Arc Pro B60 and Battlematrix

Intel NUC with NPU — the edge node

Intel software — the relevant stack

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

Pattern 2 — guardrails + PII redaction on a near-edge NUC

Pattern 3 — speculative decoding drafter on a NUC

Pattern 4 — NUC dev workstations

Unified observability in a heterogeneous cluster

Capability routing — from the L7 router to the heterogeneous one

Specific pitfalls

Applied to a generic on-premise cluster

What we have not covered (upcoming posts)

See also

References

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

TL;DR

You are here: OBSERVE (the other half of tracing)

The analogy: the cockpit of a modern aircraft

Why nvidia-smi GPU-Util misleads with LLMs

How they are wired up in production

The twelve DCGM metrics organised by family

Family 1 — Compute

Family 2 — Memory

Family 3 — Thermal and power

Family 4 — Health

The five inference engine metrics (vLLM)

The six alerts that must page in production

Master table: thresholds and queries

Three pitfalls that confuse the junior operator

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References

`profile` mode: one load point

The `analyze` subcommand: automatic concurrency sweep

Artefact structure of `analyze`

5. `ignore_eos` and an inconsistent OSL

Why `nvidia-smi` `GPU-Util` misleads with LLMs