Ai-Gateway on lo0 — Tech Blog

LLM Guard: the sworn translator with a notebook of equivalences — anatomy, scanners and its integration with Langfuse, vLLM and LiteLLM

Mon, 01 Jun 2026 05:00:00 +0200

This post is a deep dive into a single piece inside the layer covered in the post on guardrails and LLM safety. That one maps the four lines of defence (input, retrieval, tool, output) and the 2026 OSS catalogue from a bird’s-eye view; this one gets down to ground level on LLM Guard, because its Anonymize/Deanonymize pattern, its composable scanner model and its four deployment modes deserve their own treatment. The analogies built up there (HACCP kitchen, four CCPs) still hold: this post zooms in on the tool that occupies the PII belt and the individual scanners within that architecture.

TL;DR

LLM Guard is the OSS tool (MIT, Protect AI) that materialises the LLM guardrail layer with a model radically different from that of NeMo Guardrails and Guardrails AI: instead of a declarative DSL (Colang) or a validator framework with external LLM-as-judge calls, it offers a catalogue of specialised compact detectors, 15 input scanners and 21 output scanners, composable as a Python pipeline, with one distinctive mechanism: the Anonymize → LLM → Deanonymize with Vault pattern. The Vault is a centralised store of the mapping between real entities (John Doe, 12345678X) and placeholders ([REDACTED_PERSON_1], [REDACTED_DNI_1]); on input, entities are redacted and the mapping is saved; the LLM never sees real personal data; on output, the Deanonymize scanner restores the originals before returning the answer to the user. This post takes apart: the internal anatomy (Vault + scanners + orchestrator with fail_fast and TTL cache), the four deployment patterns with their mathematics (in-process library, FastAPI API, OTel sidecar over vLLM, AI Gateway plugin — LiteLLM, Envoy AI Gateway, Kong AI Gateway), the integration diagrams with Langfuse (via LLM Guard’s OTel HTTP exporter + langfuse.score() from the AI Gateway), the mathematics with the project’s benchmarks (Anonymize at 177 ms CPU → 128 ms ONNX-CPU → 125 ms GPU FP16 → 38 ms GPU+ONNX, scaling ×4.6 when you combine ONNX + GPU), the ONNX pattern as the default acceleration without a dedicated GPU, the comparison with NeMo Guardrails (declarative Colang DSL oriented to conversational flow) and Guardrails AI (JSON contract-style validators with external judges), the application to on-premise hardware (which scanners hold up on CPU, which need a shared GPU) and the seven operational pitfalls specific to the tool.

The analogy: the sworn translator with a notebook of equivalences

A serious sworn translator working with sensitive documents, an employment contract, a medical record, a tax return, does not send the raw text to the machine translation service in the cloud. They keep a notebook of equivalences open on the desk. When the original document arrives, they open the notebook and start writing down: “Marta García” → [PERSON-1], “12345678X” → [DNI-1], “ES91 2100 0418…” → [IBAN-1]. They substitute each occurrence in the text with its label and pass the anonymised text to the translation service. The service returns a translation that still contains the labels. The translator opens the notebook again, restores each label with its original value, and hands the client the final translation with the PII intact. As far as the translation service is concerned, those personal data never existed: it only saw placeholders.

This is the exact operation that defines LLM Guard’s character against the rest of the ecosystem. NeMo Guardrails solves safety with a declarative graph of rules in Colang; Guardrails AI with validators that invoke an LLM-as-judge to verify contracts; LLM Guard with a catalogue of specialised compact detectors + the Vault pattern. All three are valid in different scenarios. The choice is not a matter of taste: it is structural, according to how the system is built and where the bottleneck is.

The translator also checks, of course, that the text contains no other problems besides PII: insults, instructions to reprogram itself, links to hostile pages, code that should not be there. That is what the rest of the scanner catalogue is for. But the house signature, what sets it apart, is that notebook.

Internal anatomy of LLM Guard

The three structural pieces are:

1. The orchestrator (scan_prompt, scan_output). It receives an ordered list of scanners and runs them sequentially over the text. It returns the triple (sanitized_text, results_valid, results_score) where:

sanitized_text is the text transformed by the scanners that mutate it (Anonymize, BanSubstrings with redaction).
results_valid is a {scanner_name: bool} dict indicating which scanners passed.
results_score is a {scanner_name: float} dict with the reported risk score (0 clean, 1 maximum violation).

It supports fail_fast=True to stop after the first failure. It supports a per-scanner timeout so it does not block on a slow detector. When exposed as a FastAPI API, it supports a TTL cache to avoid rescanning repeated prompts (the case of bots with identical questions).

2. The scanner catalogue. Fifteen input scanners and twenty-one output scanners, each with its own backend model and its configurable threshold:

Family	Input	Output	Dominant backend
PII	Anonymize	Deanonymize, Sensitive	Presidio + BERT-NER
Injection and jailbreak	PromptInjection	—	Fine-tuned DeBERTa (Protect AI’s own)
Toxicity and bias	Toxicity, Sentiment	Toxicity, Bias, Sentiment	Fine-tuned RoBERTa / BERT
Banned topics	BanTopics, BanCompetitors	BanTopics, BanCompetitors	Zero-shot BART-MNLI classifier
Substrings and regex	BanSubstrings, Regex	BanSubstrings, Regex	string matching + regex
Secrets	Secrets	—	detect-secrets (Yelp) + regex
Structure	TokenLimit, Language, InvisibleText, Gibberish	JSON, Language, LanguageSame, Gibberish, ReadingTime	tokeniser, lang-detect, JSON schema
Code	BanCode, Code	BanCode, Code	language classifier + regex
URLs	—	MaliciousURLs, URLReachability	block list + DNS lookup
Answer quality	—	NoRefusal, Relevance, FactualConsistency	NLI cross-encoder + cosine similarity

Each scanner is imported and instantiated individually, with its own threshold:

from llm_guard.input_scanners import Anonymize, PromptInjection, Toxicity, Secrets
from llm_guard.vault import Vault

vault = Vault()
scanners = [
 Anonymize(vault, threshold=0.5),
 PromptInjection(threshold=0.85),
 Toxicity(threshold=0.7),
 Secrets(),
]

3. The Vault. A unique piece not found in NeMo Guardrails or Guardrails AI with the same model. It is an in-memory dictionary per session or request that stores the placeholder → original_value mapping. The Anonymize scanner writes it on input and the Deanonymize scanner reads it on output. If the Vault is shared across multiple requests from the same user, the mapping persists (useful for multi-turn conversations). If it is per request, it is discarded after the answer.

The basic Vault is a Python dict; for distributed environments with multiple pods, it is replaced by a sticky Redis (same user → same pod) or by a custom Vault that reads from and writes to an external Redis, discarded after a TTL. This is operational, not part of the core library.

The Anonymize → LLM → Deanonymize flow in detail

The canonical usage pattern of LLM Guard breaks down into six exact steps:

1. Receive the user's prompt:
"My name is Marta García and my IBAN is ES9121000418450200051332,
can you review the charge from 14 March?"
2. scan_prompt() with [Anonymize(vault), PromptInjection(), Toxicity()]
→ Anonymize redacts entities and stores them in the vault:
vault["[REDACTED_PERSON_1]"] = "Marta García"
vault["[REDACTED_IBAN_1]"] = "ES9121000418450200051332"
→ PromptInjection checks there is no jailbreak (there is none)
→ Toxicity checks there are no insults (there are none)
→ results_valid = {Anonymize: True, PromptInjection: True, Toxicity: True}
→ sanitized_prompt:
"My name is [REDACTED_PERSON_1] and my IBAN is [REDACTED_IBAN_1],
can you review the charge from 14 March?"
3. Call the LLM with sanitized_prompt:
→ vLLM receives the prompt with no real PII
→ generates the answer:
"Yes, [REDACTED_PERSON_1], I am going to review the charge on account
[REDACTED_IBAN_1]. Can you confirm the amount?"
4. scan_output() with [Deanonymize(vault), Toxicity(), Relevance(), Sensitive()]
→ Deanonymize replaces placeholders with values from the vault:
[REDACTED_PERSON_1] → "Marta García"
[REDACTED_IBAN_1] → "ES9121000418450200051332"
→ Toxicity checks the answer is not offensive
→ Relevance checks it answers the prompt
→ Sensitive checks no unauthorised PII appears
(in this case, the restored PII is authorised because the user
brought it themselves and the Vault signs it → the rule applies
only to new PII invented by the LLM)
→ sanitized_response:
"Yes, Marta García, I am going to review the charge on account
ES9121000418450200051332. Can you confirm the amount?"
5. Return sanitized_response to the user.
6. If the session continues, the vault persists and the next turns reuse
the same placeholders. When the session ends, the vault is discarded.

Three details that matter operationally:

Persistent entities ([REDACTED_PERSON_1] for “Marta García”) stay constant throughout the session. If the user mentions another person (“I spoke to Juan Pérez”), Anonymize will assign [REDACTED_PERSON_2]. Cross-turn coherence is guaranteed by the Vault.
The LLM never sees the original data during the session. This is the key property for cases where the LLM is served from a cloud model or when the prompt is logged (Langfuse, OTel) without confidential access.
LLM Guard’s logging records the placeholders, not the original values. For audit with original values, an additional layer is needed (access to the Vault with privileged permissions), and this is by design, not by default.

Four deployment modes

Mode 1 — In-process Python library

The simplest: pip install llm-guard, import the scanners in the application code, call scan_prompt/scan_output directly. The models load into the process. The advantage is minimum latency; the disadvantage is that each application replica loads its own models into memory.

# on the app server
from llm_guard import scan_prompt, scan_output
from llm_guard.input_scanners import Anonymize, PromptInjection, Toxicity
from llm_guard.output_scanners import Deanonymize, Toxicity as OutToxicity, Relevance
from llm_guard.vault import Vault

vault = Vault()
input_scanners = [Anonymize(vault), PromptInjection(), Toxicity()]
output_scanners = [Deanonymize(vault), OutToxicity(), Relevance()]

# in the request handler
sanitized_prompt, valid_in, score_in = scan_prompt(input_scanners, user_prompt)
if not all(valid_in.values()):
 return error_response(score_in)

response = vllm_client.complete(sanitized_prompt)

sanitized_resp, valid_out, score_out = scan_output(output_scanners, sanitized_prompt, response)
if not all(valid_out.values()):
 return error_response(score_out)

return sanitized_resp

It fits pattern A (sidecar) from the guardrails post when the app and the sidecar share a process. And pattern C (in-process) if the app is itself the inference layer.

Mode 2 — Its own FastAPI API

The project includes a ready-made FastAPI server (llm-guard-api) that exposes the scanners behind two REST endpoints:

POST /analyze/prompt
body: {"prompt": "...", "scanners": [...] (optional)}
response: {"sanitized_prompt": "...", "is_valid": bool, "scanners": {scanner: {is_valid, risk_score}}}
POST /analyze/output
body: {"prompt": "...", "output": "...", "scanners": [...]}
response: analogous

Configuration through config/scanners.yml with environment variables (SCAN_FAIL_FAST, CACHE_MAX_SIZE, CACHE_TTL, SCAN_PROMPT_TIMEOUT…). It carries Prometheus metrics at /metrics and OTel HTTP exporter traces by default.

It fits pattern B (centralised service behind an AI gateway) from the guardrails post.

Mode 3 — OTel sidecar on the inference engine pod

For vLLM deployments on Kubernetes, a variant of mode 2 is to deploy the LLM Guard API as a sidecar container in the same vLLM pod, talking over localhost. The AI gateway in front invokes the sidecar before and after inference. The node’s OTel collector aggregates vLLM’s spans with LLM Guard’s gen_ai.guardrail.* spans automatically because they share a trace_id propagated via HTTP baggage.

This fits pattern A (sidecar) from the guardrails post, but with the discipline of a REST API so as not to couple languages (the AI gateway can be Envoy in C++, LLM Guard in Python).

Mode 4 — Plugin inside an AI gateway

Three AI gateways support LLM Guard as a native plugin in 2026:

LiteLLM Proxy (MIT, BerriAI) — llm_guard plugin, enabled in config with guardrails: ["llm_guard"]. It calls the API internally.
Envoy AI Gateway (CNCF, Apache 2.0) — ai-guardrails filter with a pluggable backend pointing at the LLM Guard service.
Kong AI Gateway (Apache 2.0) — ai-proxy plugin with a post-processor that invokes LLM Guard.

In all three cases, the AI gateway is the single entry point from the client app to the LLM; the gateway calls LLM Guard before and after passing to the inference engine. Advantage: zero lock-in in the application code; switching from LLM Guard to NeMo Guardrails means changing the gateway plugin, not rewriting the app. Disadvantage: the extra hop adds latency (typically 5-15 ms intra-cluster).

Graphical integration with Langfuse, vLLM and the OTel stack

The three integration routes with Langfuse that matter operationally:

Route A — LLM Guard’s OTel HTTP exporter. LLM Guard has a native OTel HTTP exporter. By setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://langfuse.cluster/api/public/otel, the gen_ai.guardrail.* spans emitted by each scanner arrive directly at Langfuse and appear as child spans of the main LLM span (provided the trace_id is propagated via HTTP baggage from the AI gateway). This is the canonical route in 2026.

Route B — Langfuse scoring API from the AI gateway. On receiving LLM Guard’s response with the per-scanner risk_score, the AI gateway (LiteLLM, Envoy AI, Kong AI) issues a langfuse.score(trace_id, name="guardrail.PromptInjection", value=0.87, comment="blocked") call for each scanner. In Langfuse it appears as scores attached to the same trace as the inference. It enables “blocks per category” dashboards and per-scanner time series. It is complementary to route A: A brings the spans, B brings the numeric score that is easy to aggregate in SQL.

Route C — Langfuse sessions + Vault metadata. In conversational mode, the AI gateway propagates langfuse_session_id to the Vault as its key. When a user has a multi-turn session, Langfuse shows the complete trace of the session, with the placeholders reused turn after turn. The original PII still never travels to Langfuse, only the placeholders and their categories.

The node’s OTel Collector is the glue: it receives spans from vLLM (via OpenLLMetry or native instrumentation), from LLM Guard (via its OTel exporter) and from the AI gateway (standard HTTP instrumentation), joins them by trace_id, and sends them in parallel to Langfuse (via OTLP HTTP) and to Tempo/Jaeger. LLM Guard’s Prometheus metrics go to VictoriaMetrics via normal scraping. Grafana offers the unified view for cross-trace investigation; Langfuse offers the LLM-centric view with sessions and scores. The OTel GenAI tracing post details the complete mechanics of the Collector.

The mathematics that matter

Latency per scanner — the real numbers

The project publishes reproducible benchmarks. For the Anonymize scanner (input length 317 chars, batch 5), the reference data are:

Platform	Backend	Avg latency	p99	QPS
AWS m5.xlarge (CPU)	Transformers	177 ms	326 ms	1,789
AWS m5.xlarge (CPU)	ONNX runtime	128 ms	180 ms	2,464
AWS r6a.xlarge (AMD CPU)	Transformers	244 ms	284 ms	1,298
AWS g5.xlarge (NVIDIA A10G)	Transformers FP16	125 ms	498 ms	2,532
AWS g5.xlarge (A10G)	ONNX + GPU	38 ms	99 ms	8,317

Three operational observations:

ONNX always wins. Even on CPU, ONNX brings the average down from 177 to 128 ms (a factor of 1.4×). On GPU with ONNX, it drops from 177 to 38 ms (a factor of 4.6×). The practical rule: always export the scanner’s model to ONNX before production. The official SaaS preview uses it by default.
A GPU without ONNX does not perform as well as you might expect. An A10G without ONNX (125 ms) is comparable to an m5.xlarge with ONNX (128 ms). The GPU alone does not compensate if the inference graph is not optimised. The relevant pairing is ONNX + GPU.
p99 latency without ONNX explodes. On GPU without ONNX, the p99 of 498 ms triples the average of 125 ms — queues and batching produce high tail latencies. With ONNX, the p99/avg ratio drops to 2.6× (99/38), far more predictable.

For a guardrail layer with five scanners run sequentially (Anonymize, PromptInjection, Toxicity, Secrets, BanTopics), the sum of the p99s is what determines the budget for line 1 (input). Five scanners at ~100 ms p99 each = 500 ms accumulated p99, outside the budget for interactive chat. With ONNX we get down to ~50 ms each = 250 ms p99, which is manageable. With fail_fast=True, the expected time is lower (the most likely case is that the cheap ones pass and the expensive ones only fail if they run at all).

For a finer calculation, the expected latency of the pipeline with fail_fast is:

\[ \mathbb{E}[L] = \sum_{i=1}^{N} L_i \cdot \prod_{j=1}^{i-1} p_j \]

where \(L_i\) is the latency of scanner \(i\) and \(p_j\) the probability that scanner \(j\) returns valid. On well-behaved traffic (most prompts pass every scanner), \(\prod p_j \approx 1\) and the formula collapses to the direct sum. On adversarial traffic, the faster scanners at the start of the pipeline cut in earlier and the expected latency drops sharply.

Computational cost per scanner

The size of the backend model determines the cost and the possibility of running on CPU vs requiring a GPU:

Scanner	Typical backend model	Parameters	VRAM FP16 / ONNX-INT8	CPU viable
Anonymize (BERT-NER)	dslim/bert-base-NER	110 M	220 MB / 55 MB	Yes (with ONNX)
Anonymize (BERT-large)	dslim/bert-large-NER	335 M	670 MB / 170 MB	Yes but slow (~500 ms CPU)
PromptInjection	Fine-tuned DeBERTa-v3-base	184 M	370 MB / 90 MB	Yes (with ONNX)
Toxicity	unitary/toxic-bert	110 M	220 MB / 55 MB	Yes
Sentiment	distilbert-sst2	67 M	130 MB / 35 MB	Yes
Gibberish	small distilbert	67 M	130 MB / 35 MB	Yes
BanTopics	BART-MNLI zero-shot	407 M	815 MB / 200 MB	Slow on CPU (~400 ms)
Bias (output)	RoBERTa-bias	125 M	250 MB / 65 MB	Yes
FactualConsistency	cross-encoder/nli-deberta	184 M	370 MB / 90 MB	Yes
Relevance	sentence-transformers	110 M	220 MB / 55 MB	Yes
TokenLimit, Regex, JSON, BanSubstrings, Secrets	(no model)	—	0	Trivial

Sensible on-premise pattern: model-free scanners (TokenLimit, Regex, BanSubstrings, Secrets) run on CPU without blinking. Anonymize, PromptInjection, Toxicity, Sentiment and Relevance run comfortably on CPU with ONNX-INT8 at ~50-150 ms p99. BanTopics and the ones based on large cross-encoders (FactualConsistency) are the candidates to live on a shared GPU if you want p99 < 100 ms.

API throughput on a cluster

One instance of the FastAPI API with 4 Uvicorn workers on a node with 8 vCPUs reaches ~600-1,200 RPS over a typical 5-scanner pipeline on CPU + ONNX. To scale:

Horizontally: replicate pods behind a ClusterIP Service — linear scaling, because the scanners are stateless (except the Vault, which is per session and is externalised to Redis if you want it sticky or shared).
Vertically with GPU: 1 H100 serves ~5,000-10,000 RPS with all the scanners in ONNX-GPU. It is overkill for most deployments except multi-tenant ones with thousands of sustained QPS.

The practical rule from the guardrails post (1 guardrail GPU per 4-8 LLM GPUs) holds here: with a 4×H100 SXM cluster serving Llama 70B at TP=4, an L4 or RTX 4090 dedicated to the LLM Guard service covers the load.

Comparison with NeMo Guardrails and Guardrails AI

The three tools solve the same problem from three different architectural models. The choice between them is not about quality, since all three are mature, it is about fit with the rest of the stack:

Dimension	LLM Guard	NeMo Guardrails	Guardrails AI
Conceptual model	Pipeline of compact scanners	Declarative Colang graph (conversational flow)	JSON contract-style validators
Dominant detection	Specialised ML models (BERT, DeBERTa) per category	Rules + LLM-as-judge	Heuristic validators + external LLM-as-judge
PII workflow	Anonymize + Vault + Deanonymize	Via integrated Presidio, no built-in Vault	PII validators, no automatic restitution
Licence	MIT	Apache 2.0	Apache 2.0 (+ paid Hub)
Language	Python	Python + Colang DSL	Python
API maturity	Built-in FastAPI API, built-in OTel	Built-in FastAPI server, partial OTel	External API server
Cluster deployment	Lib + API + sidecar + gateway plugins	Lib + server	Lib + server + SaaS Hub
Typical latency (5 ONNX-GPU scanners)	50-200 ms	100-500 ms (more if there is an LLM judge)	100-300 ms (depends on the validator)
When it shines	Apps with heavy PII, multi-tenant with sessions, GDPR/HIPAA requirements	Conversational systems with defined flows, agents with a dialogue policy	Apps with strict JSON contracts, structured output with additional validation
When it does not fit	If you need a declarative dialogue policy	If you want compact detectors with no LLM judge	If you want an automatic Vault and Deanonymize

The three are complementary in large deployments. A mature pattern in 2026 is:

NeMo Guardrails orchestrates the dialogue flow (which tools the agent can invoke, in what order, with what cooldowns).
LLM Guard occupies the PII line + compact scanners on input and output, with its Vault doing the dirty work of anonymisation.
Guardrails AI validates structured outputs (JSON Schema, function calling) with its validators.

The separation of responsibilities avoids overlap and allows pieces to be swapped without rewriting everything. All three expose a FastAPI API and emit OTel spans; the AI gateway orchestrates them sequentially.

Applied to on-premise hardware

On the RTX 4090 (24 GB)

A 4090 dedicated to the LLM Guard service pod comfortably serves the complete pipeline in medium-scale production:

Anonymize (BERT-NER ONNX-INT8): ~50 MB VRAM.
PromptInjection (DeBERTa ONNX-INT8): ~90 MB.
Toxicity, Sentiment, Gibberish: ~150 MB total.
BanTopics (BART-MNLI ONNX-INT8): ~200 MB.
Bias, Relevance, FactualConsistency (output): ~250 MB total.

Total ~750 MB. The rest of the VRAM is idle or usable for aggressive batching. Sustained throughput of 3,000-6,000 RPS over the complete pipeline. For deployments with < 500 sustained RPS, the 4090 is under-used and can be shared with another workload (RAG embeddings, BGE reranker).

On the 4×H100 SXM cluster (320 GB total, NVLink)

There is capacity to spare by an order of magnitude. Sensible pattern:

3 H100s serving the main LLM at TP=3 (Llama 70B FP8).
1 H100 split into MIG instances (1g.10gb or similar) — one slice for LLM Guard (~10 GB MIG is more than enough), another for the reranker, another for embeddings.

Aggregate throughput for LLM Guard at that scale: 15,000-30,000 RPS. Plenty for a large multi-tenant setup with long sessions.

The specific operational pitfalls

Pitfall 1 — Vault with no TTL. The Vault grows without limit if it is not cleaned. In in-process lib mode per request there is no problem (the object is destroyed). In centralised service mode with Redis, the TTL is missing and Redis fills up. A silent trap discovered when the Redis pod gets OOM-killed in production after six weeks.

Pitfall 2 — Vault not shared between pods + AI gateway with no sticky session. If the AI gateway distributes round-robin across multiple LLM Guard pods, one pod’s local Vault knows nothing of the mapping created by another. Result: on turn 2 of a session, Deanonymize does not find the placeholders from turn 1 and leaves a literal [REDACTED_PERSON_1] in the answer. Solution: a shared Redis Vault or a sticky session by user_id.

Pitfall 3 — Models not exported to ONNX in production. It is deployed with the default config (Transformers) and latency is 3-5× worse than the benchmarks report. The team assumes LLM Guard “is slow”. The solution is to export to ONNX (built into the project) and configure recognizer_conf with the path to the model’s .onnx.

Pitfall 4 — fail_fast=False with many scanners. Without fail_fast, every scanner always runs, even if the first one already blocked. Latency is 3-5× worse on adversarial traffic. For production, barring an explicit reason (wanting complete per-scanner metrics even when blocking), fail_fast=True is the reasonable default.

Pitfall 5 — Infinite cache_ttl + prompts with variable PII. If the API’s cache stores the sanitized_prompt indefinitely, two different sessions with the same prompt structure but different PII can collide if the cache key does not include the Vault hash. You have to verify that the cache key includes either the complete content (without PII) or a hash of the original prompt.

Pitfall 6 — Structured logs with original PII. LLM Guard’s JSON stdout logs record only placeholders by default. But if custom hooks are added for debugging, it is easy to leak the original PII to the log. A regulatory audit (GDPR, ENS) detects this and it is a breach. Discipline: never add hooks that read from the Vault without explicit permission.

Pitfall 7 — scan_output without the original prompt. The scan_output method expects (prompt, output) for validators that compare both (Relevance, LanguageSame, FactualConsistency). If only the output is passed, those scanners fail silently or return is_valid=True by default. You have to keep the sanitized_prompt in the AI gateway and pass it to scan_output.

When to choose LLM Guard (and when not to)

Choose LLM Guard when:

The requirement for PII anonymisation with automatic restitution is on the list. It is the number one reason to use it. Banking, healthcare, legal advice, HR — any case with heavy PII that must not reach the LLM even when it is local.
You want a Pythonic pipeline with no new DSL. If the team is pure Python and prefers composing scanners as objects to learning Colang.
The stack already has an AI gateway (LiteLLM, Envoy AI, Kong AI) and it integrates as a plugin without touching the app.
You need built-in OTel and Prometheus with no additional instrumentation.

Do not choose LLM Guard when:

The system is a conversational agent with complex dialogue flows (policies, fallbacks, escalation to a human). There NeMo Guardrails with Colang is structurally better.
The safety layer reduces to validating structured outputs (JSON, function calling). Guardrails AI with its validators is more natural.
Your latency budget is ultra-aggressive (< 30 ms for the whole layer). You will have to cut scanners and accept lower coverage; perhaps a single PromptGuard 2 + Presidio in a sidecar (the pattern from the guardrails post) is simpler.
You do not want to carry the operational weight of a distributed Vault (Redis, TTL, sticky session). For small systems with no strong PII requirement, it is over-engineered.

What we have not covered (upcoming posts)

Custom scanners: how to write your own scanner when nothing in the catalogue fits (a complex domain regex, your own fine-tuned classifier). The project supports custom scanners by inheriting from InputScanner / OutputScanner with three methods.
Integration with SLSA / supply chain: how to sign the LLM Guard container with cosign, SLSA attestations, and verification in the cluster before admitting it. An operational supply chain security topic (OWASP LLM03).
Red teaming against LLM Guard: known techniques that evade detectors (homoglyphs, Unicode confusables, base64 encoding inside the prompt). The project publishes an adversarial test suite for doing your own benchmarking. How it is set up as a continuous gate in CI.
Comparative benchmark against Bedrock Guardrails and Azure AI Content Safety: F1 per category over real traffic across three different deployments. The OSS vs hyperscalers post has the strategic comparison; the technical detection comparison is still missing.

References

LLM Guard (Protect AI): https://llm-guard.com — official documentation, scanner list, benchmarks.
Repository: https://github.com/protectai/llm-guard.
LLM Guard API: https://github.com/protectai/llm-guard/tree/main/llm_guard_api.
Presidio (Microsoft): https://microsoft.github.io/presidio/ — basis of the Anonymize scanner.
detect-secrets (Yelp): https://github.com/Yelp/detect-secrets — basis of the Secrets scanner.
Langfuse OTel ingestion: https://langfuse.com/docs/opentelemetry/get-started.
LiteLLM guardrails: https://docs.litellm.ai/docs/proxy/guardrails.
Envoy AI Gateway: https://aigateway.envoyproxy.io.
Kong AI Gateway: https://docs.konghq.com/hub/kong-inc/ai-prompt-guard/.
OWASP Top 10 for LLM Applications 2025: https://owasp.org/www-project-top-10-for-large-language-model-applications/.
ONNX Runtime: https://onnxruntime.ai — exporting HF models to ONNX for acceleration.

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

Ai-Gateway on lo0 — Tech Blog

LLM Guard: the sworn translator with a notebook of equivalences — anatomy, scanners and its integration with Langfuse, vLLM and LiteLLM

TL;DR

The analogy: the sworn translator with a notebook of equivalences

Internal anatomy of LLM Guard

The Anonymize → LLM → Deanonymize flow in detail

Four deployment modes

Mode 1 — In-process Python library

Mode 2 — Its own FastAPI API

Mode 3 — OTel sidecar on the inference engine pod

Mode 4 — Plugin inside an AI gateway

Graphical integration with Langfuse, vLLM and the OTel stack

The mathematics that matter

Latency per scanner — the real numbers

Computational cost per scanner

API throughput on a cluster

Comparison with NeMo Guardrails and Guardrails AI

Applied to on-premise hardware

On the RTX 4090 (24 GB)

On the 4×H100 SXM cluster (320 GB total, NVLink)

The specific operational pitfalls

When to choose LLM Guard (and when not to)

What we have not covered (upcoming posts)

References

See also

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

TL;DR

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

The four layers of a multi-tenant inference platform

Layer 1 — AI Gateway: the single front door

Why centralise

The three dominant options in 2026

When to choose each one

The minimum the gateway has to do

Example of a multi-tenant LiteLLM configuration

Layer 2 — Policy & Quota Plane: what each tenant can do

Technical quotas

Economic budgets

Model whitelist and blacklist

Priority classes

Admission control

The typical pattern in 2026

Layer 3 — Isolation Plane: isolating workloads physically

Three NVIDIA mechanisms for sharing a GPU

The choice for multi-tenant in 2026

MIG partitions on the H100

Approach A — A shared large model with quotas at the gateway

Approach B — Dedicating GPUs per model / tenant

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

The operational choice: start with A, move up to C if you need to

Isolation at the Kubernetes level

Layer 4 — Observability Plane: seeing what happens per tenant

The four mandatory properties

Showback vs chargeback

Tools

The minimum multi-tenant dashboard

Sizing on mid-scale GPU clusters: concrete decisions

Default decisions

An example tenant scheme

When to add hardware

Common operational traps

A gateway without auth: a backdoor into the cluster

MIG and NVLink are incompatible

Quotas pinned to the cluster’s ceiling

No multi-tenant observability from day 1

Showback that never reaches chargeback

Non-whitelisted models eating the budget

Badly calibrated priority classes

No failover from the gateway

An operational roadmap for getting started

What we have not covered (upcoming posts)

References