LLM Guard: the sworn translator with a notebook of equivalences — anatomy, scanners and its integration with Langfuse, vLLM and LiteLLM

This post is a deep dive into a single piece inside the layer covered in the post on guardrails and LLM safety. That one maps the four lines of defence (input, retrieval, tool, output) and the 2026 OSS catalogue from a bird’s-eye view; this one gets down to ground level on LLM Guard, because its Anonymize/Deanonymize pattern, its composable scanner model and its four deployment modes deserve their own treatment. The analogies built up there (HACCP kitchen, four CCPs) still hold: this post zooms in on the tool that occupies the PII belt and the individual scanners within that architecture.

TL;DR

LLM Guard is the OSS tool (MIT, Protect AI) that materialises the LLM guardrail layer with a model radically different from that of NeMo Guardrails and Guardrails AI: instead of a declarative DSL (Colang) or a validator framework with external LLM-as-judge calls, it offers a catalogue of specialised compact detectors, 15 input scanners and 21 output scanners, composable as a Python pipeline, with one distinctive mechanism: the Anonymize → LLM → Deanonymize with Vault pattern. The Vault is a centralised store of the mapping between real entities (John Doe, 12345678X) and placeholders ([REDACTED_PERSON_1], [REDACTED_DNI_1]); on input, entities are redacted and the mapping is saved; the LLM never sees real personal data; on output, the Deanonymize scanner restores the originals before returning the answer to the user. This post takes apart: the internal anatomy (Vault + scanners + orchestrator with fail_fast and TTL cache), the four deployment patterns with their mathematics (in-process library, FastAPI API, OTel sidecar over vLLM, AI Gateway plugin — LiteLLM, Envoy AI Gateway, Kong AI Gateway), the integration diagrams with Langfuse (via LLM Guard’s OTel HTTP exporter + langfuse.score() from the AI Gateway), the mathematics with the project’s benchmarks (Anonymize at 177 ms CPU → 128 ms ONNX-CPU → 125 ms GPU FP16 → 38 ms GPU+ONNX, scaling ×4.6 when you combine ONNX + GPU), the ONNX pattern as the default acceleration without a dedicated GPU, the comparison with NeMo Guardrails (declarative Colang DSL oriented to conversational flow) and Guardrails AI (JSON contract-style validators with external judges), the application to on-premise hardware (which scanners hold up on CPU, which need a shared GPU) and the seven operational pitfalls specific to the tool.

The analogy: the sworn translator with a notebook of equivalences

A serious sworn translator working with sensitive documents, an employment contract, a medical record, a tax return, does not send the raw text to the machine translation service in the cloud. They keep a notebook of equivalences open on the desk. When the original document arrives, they open the notebook and start writing down: “Marta García” → [PERSON-1], “12345678X” → [DNI-1], “ES91 2100 0418…” → [IBAN-1]. They substitute each occurrence in the text with its label and pass the anonymised text to the translation service. The service returns a translation that still contains the labels. The translator opens the notebook again, restores each label with its original value, and hands the client the final translation with the PII intact. As far as the translation service is concerned, those personal data never existed: it only saw placeholders.

This is the exact operation that defines LLM Guard’s character against the rest of the ecosystem. NeMo Guardrails solves safety with a declarative graph of rules in Colang; Guardrails AI with validators that invoke an LLM-as-judge to verify contracts; LLM Guard with a catalogue of specialised compact detectors + the Vault pattern. All three are valid in different scenarios. The choice is not a matter of taste: it is structural, according to how the system is built and where the bottleneck is.

The translator also checks, of course, that the text contains no other problems besides PII: insults, instructions to reprogram itself, links to hostile pages, code that should not be there. That is what the rest of the scanner catalogue is for. But the house signature, what sets it apart, is that notebook.

Internal anatomy of LLM Guard

The three structural pieces are:

1. The orchestrator (scan_prompt, scan_output). It receives an ordered list of scanners and runs them sequentially over the text. It returns the triple (sanitized_text, results_valid, results_score) where:

• sanitized_text is the text transformed by the scanners that mutate it (Anonymize, BanSubstrings with redaction).
• results_valid is a {scanner_name: bool} dict indicating which scanners passed.
• results_score is a {scanner_name: float} dict with the reported risk score (0 clean, 1 maximum violation).

It supports fail_fast=True to stop after the first failure. It supports a per-scanner timeout so it does not block on a slow detector. When exposed as a FastAPI API, it supports a TTL cache to avoid rescanning repeated prompts (the case of bots with identical questions).

2. The scanner catalogue. Fifteen input scanners and twenty-one output scanners, each with its own backend model and its configurable threshold:

Each scanner is imported and instantiated individually, with its own threshold:

from llm_guard.input_scanners import Anonymize, PromptInjection, Toxicity, Secrets
from llm_guard.vault import Vault

vault = Vault()
scanners = [
    Anonymize(vault, threshold=0.5),
    PromptInjection(threshold=0.85),
    Toxicity(threshold=0.7),
    Secrets(),
]

3. The Vault. A unique piece not found in NeMo Guardrails or Guardrails AI with the same model. It is an in-memory dictionary per session or request that stores the placeholder → original_value mapping. The Anonymize scanner writes it on input and the Deanonymize scanner reads it on output. If the Vault is shared across multiple requests from the same user, the mapping persists (useful for multi-turn conversations). If it is per request, it is discarded after the answer.

The basic Vault is a Python dict; for distributed environments with multiple pods, it is replaced by a sticky Redis (same user → same pod) or by a custom Vault that reads from and writes to an external Redis, discarded after a TTL. This is operational, not part of the core library.

The Anonymize → LLM → Deanonymize flow in detail

The canonical usage pattern of LLM Guard breaks down into six exact steps:

1. Receive the user's prompt:
   "My name is Marta García and my IBAN is ES9121000418450200051332,
    can you review the charge from 14 March?"

2. scan_prompt() with [Anonymize(vault), PromptInjection(), Toxicity()]
   → Anonymize redacts entities and stores them in the vault:
       vault["[REDACTED_PERSON_1]"] = "Marta García"
       vault["[REDACTED_IBAN_1]"] = "ES9121000418450200051332"
   → PromptInjection checks there is no jailbreak (there is none)
   → Toxicity checks there are no insults (there are none)
   → results_valid = {Anonymize: True, PromptInjection: True, Toxicity: True}
   → sanitized_prompt:
   "My name is [REDACTED_PERSON_1] and my IBAN is [REDACTED_IBAN_1],
    can you review the charge from 14 March?"

3. Call the LLM with sanitized_prompt:
   → vLLM receives the prompt with no real PII
   → generates the answer:
   "Yes, [REDACTED_PERSON_1], I am going to review the charge on account
    [REDACTED_IBAN_1]. Can you confirm the amount?"

4. scan_output() with [Deanonymize(vault), Toxicity(), Relevance(), Sensitive()]
   → Deanonymize replaces placeholders with values from the vault:
       [REDACTED_PERSON_1] → "Marta García"
       [REDACTED_IBAN_1]   → "ES9121000418450200051332"
   → Toxicity checks the answer is not offensive
   → Relevance checks it answers the prompt
   → Sensitive checks no unauthorised PII appears
     (in this case, the restored PII is authorised because the user
      brought it themselves and the Vault signs it → the rule applies
      only to new PII invented by the LLM)
   → sanitized_response:
   "Yes, Marta García, I am going to review the charge on account
    ES9121000418450200051332. Can you confirm the amount?"

5. Return sanitized_response to the user.

6. If the session continues, the vault persists and the next turns reuse
   the same placeholders. When the session ends, the vault is discarded.

Three details that matter operationally:

• Persistent entities ([REDACTED_PERSON_1] for “Marta García”) stay constant throughout the session. If the user mentions another person (“I spoke to Juan Pérez”), Anonymize will assign [REDACTED_PERSON_2]. Cross-turn coherence is guaranteed by the Vault.
• The LLM never sees the original data during the session. This is the key property for cases where the LLM is served from a cloud model or when the prompt is logged (Langfuse, OTel) without confidential access.
• LLM Guard’s logging records the placeholders, not the original values. For audit with original values, an additional layer is needed (access to the Vault with privileged permissions), and this is by design, not by default.

Four deployment modes

Mode 1 — In-process Python library

The simplest: pip install llm-guard, import the scanners in the application code, call scan_prompt/scan_output directly. The models load into the process. The advantage is minimum latency; the disadvantage is that each application replica loads its own models into memory.

# on the app server
from llm_guard import scan_prompt, scan_output
from llm_guard.input_scanners import Anonymize, PromptInjection, Toxicity
from llm_guard.output_scanners import Deanonymize, Toxicity as OutToxicity, Relevance
from llm_guard.vault import Vault

vault = Vault()
input_scanners = [Anonymize(vault), PromptInjection(), Toxicity()]
output_scanners = [Deanonymize(vault), OutToxicity(), Relevance()]

# in the request handler
sanitized_prompt, valid_in, score_in = scan_prompt(input_scanners, user_prompt)
if not all(valid_in.values()):
    return error_response(score_in)

response = vllm_client.complete(sanitized_prompt)

sanitized_resp, valid_out, score_out = scan_output(output_scanners, sanitized_prompt, response)
if not all(valid_out.values()):
    return error_response(score_out)

return sanitized_resp

It fits pattern A (sidecar) from the guardrails post when the app and the sidecar share a process. And pattern C (in-process) if the app is itself the inference layer.

Mode 2 — Its own FastAPI API

The project includes a ready-made FastAPI server (llm-guard-api) that exposes the scanners behind two REST endpoints:

POST /analyze/prompt
  body: {"prompt": "...", "scanners": [...] (optional)}
  response: {"sanitized_prompt": "...", "is_valid": bool, "scanners": {scanner: {is_valid, risk_score}}}

POST /analyze/output
  body: {"prompt": "...", "output": "...", "scanners": [...]}
  response: analogous

Configuration through config/scanners.yml with environment variables (SCAN_FAIL_FAST, CACHE_MAX_SIZE, CACHE_TTL, SCAN_PROMPT_TIMEOUT…). It carries Prometheus metrics at /metrics and OTel HTTP exporter traces by default.

It fits pattern B (centralised service behind an AI gateway) from the guardrails post.

Mode 3 — OTel sidecar on the inference engine pod

For vLLM deployments on Kubernetes, a variant of mode 2 is to deploy the LLM Guard API as a sidecar container in the same vLLM pod, talking over localhost. The AI gateway in front invokes the sidecar before and after inference. The node’s OTel collector aggregates vLLM’s spans with LLM Guard’s gen_ai.guardrail.* spans automatically because they share a trace_id propagated via HTTP baggage.

This fits pattern A (sidecar) from the guardrails post, but with the discipline of a REST API so as not to couple languages (the AI gateway can be Envoy in C++, LLM Guard in Python).

Mode 4 — Plugin inside an AI gateway

Three AI gateways support LLM Guard as a native plugin in 2026:

• LiteLLM Proxy (MIT, BerriAI) — llm_guard plugin, enabled in config with guardrails: ["llm_guard"]. It calls the API internally.
• Envoy AI Gateway (CNCF, Apache 2.0) — ai-guardrails filter with a pluggable backend pointing at the LLM Guard service.
• Kong AI Gateway (Apache 2.0) — ai-proxy plugin with a post-processor that invokes LLM Guard.

In all three cases, the AI gateway is the single entry point from the client app to the LLM; the gateway calls LLM Guard before and after passing to the inference engine. Advantage: zero lock-in in the application code; switching from LLM Guard to NeMo Guardrails means changing the gateway plugin, not rewriting the app. Disadvantage: the extra hop adds latency (typically 5-15 ms intra-cluster).

Graphical integration with Langfuse, vLLM and the OTel stack

The three integration routes with Langfuse that matter operationally:

Route A — LLM Guard’s OTel HTTP exporter. LLM Guard has a native OTel HTTP exporter. By setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://langfuse.cluster/api/public/otel, the gen_ai.guardrail.* spans emitted by each scanner arrive directly at Langfuse and appear as child spans of the main LLM span (provided the trace_id is propagated via HTTP baggage from the AI gateway). This is the canonical route in 2026.

Route B — Langfuse scoring API from the AI gateway. On receiving LLM Guard’s response with the per-scanner risk_score, the AI gateway (LiteLLM, Envoy AI, Kong AI) issues a langfuse.score(trace_id, name="guardrail.PromptInjection", value=0.87, comment="blocked") call for each scanner. In Langfuse it appears as scores attached to the same trace as the inference. It enables “blocks per category” dashboards and per-scanner time series. It is complementary to route A: A brings the spans, B brings the numeric score that is easy to aggregate in SQL.

Route C — Langfuse sessions + Vault metadata. In conversational mode, the AI gateway propagates langfuse_session_id to the Vault as its key. When a user has a multi-turn session, Langfuse shows the complete trace of the session, with the placeholders reused turn after turn. The original PII still never travels to Langfuse, only the placeholders and their categories.

The node’s OTel Collector is the glue: it receives spans from vLLM (via OpenLLMetry or native instrumentation), from LLM Guard (via its OTel exporter) and from the AI gateway (standard HTTP instrumentation), joins them by trace_id, and sends them in parallel to Langfuse (via OTLP HTTP) and to Tempo/Jaeger. LLM Guard’s Prometheus metrics go to VictoriaMetrics via normal scraping. Grafana offers the unified view for cross-trace investigation; Langfuse offers the LLM-centric view with sessions and scores. The OTel GenAI tracing post details the complete mechanics of the Collector.

The mathematics that matter

Latency per scanner — the real numbers

The project publishes reproducible benchmarks. For the Anonymize scanner (input length 317 chars, batch 5), the reference data are:

Three operational observations:

1. ONNX always wins. Even on CPU, ONNX brings the average down from 177 to 128 ms (a factor of 1.4×). On GPU with ONNX, it drops from 177 to 38 ms (a factor of 4.6×). The practical rule: always export the scanner’s model to ONNX before production. The official SaaS preview uses it by default.
2. A GPU without ONNX does not perform as well as you might expect. An A10G without ONNX (125 ms) is comparable to an m5.xlarge with ONNX (128 ms). The GPU alone does not compensate if the inference graph is not optimised. The relevant pairing is ONNX + GPU.
3. p99 latency without ONNX explodes. On GPU without ONNX, the p99 of 498 ms triples the average of 125 ms — queues and batching produce high tail latencies. With ONNX, the p99/avg ratio drops to 2.6× (99/38), far more predictable.

For a guardrail layer with five scanners run sequentially (Anonymize, PromptInjection, Toxicity, Secrets, BanTopics), the sum of the p99s is what determines the budget for line 1 (input). Five scanners at ~100 ms p99 each = 500 ms accumulated p99, outside the budget for interactive chat. With ONNX we get down to ~50 ms each = 250 ms p99, which is manageable. With fail_fast=True, the expected time is lower (the most likely case is that the cheap ones pass and the expensive ones only fail if they run at all).

For a finer calculation, the expected latency of the pipeline with fail_fast is:

where \(L_i\) is the latency of scanner \(i\) and \(p_j\) the probability that scanner \(j\) returns valid. On well-behaved traffic (most prompts pass every scanner), \(\prod p_j \approx 1\) and the formula collapses to the direct sum. On adversarial traffic, the faster scanners at the start of the pipeline cut in earlier and the expected latency drops sharply.

Computational cost per scanner

The size of the backend model determines the cost and the possibility of running on CPU vs requiring a GPU:

Sensible on-premise pattern: model-free scanners (TokenLimit, Regex, BanSubstrings, Secrets) run on CPU without blinking. Anonymize, PromptInjection, Toxicity, Sentiment and Relevance run comfortably on CPU with ONNX-INT8 at ~50-150 ms p99. BanTopics and the ones based on large cross-encoders (FactualConsistency) are the candidates to live on a shared GPU if you want p99 < 100 ms.

API throughput on a cluster

One instance of the FastAPI API with 4 Uvicorn workers on a node with 8 vCPUs reaches ~600-1,200 RPS over a typical 5-scanner pipeline on CPU + ONNX. To scale:

• Horizontally: replicate pods behind a ClusterIP Service — linear scaling, because the scanners are stateless (except the Vault, which is per session and is externalised to Redis if you want it sticky or shared).
• Vertically with GPU: 1 H100 serves ~5,000-10,000 RPS with all the scanners in ONNX-GPU. It is overkill for most deployments except multi-tenant ones with thousands of sustained QPS.

The practical rule from the guardrails post (1 guardrail GPU per 4-8 LLM GPUs) holds here: with a 4×H100 SXM cluster serving Llama 70B at TP=4, an L4 or RTX 4090 dedicated to the LLM Guard service covers the load.

Comparison with NeMo Guardrails and Guardrails AI

The three tools solve the same problem from three different architectural models. The choice between them is not about quality, since all three are mature, it is about fit with the rest of the stack:

The three are complementary in large deployments. A mature pattern in 2026 is:

• NeMo Guardrails orchestrates the dialogue flow (which tools the agent can invoke, in what order, with what cooldowns).
• LLM Guard occupies the PII line + compact scanners on input and output, with its Vault doing the dirty work of anonymisation.
• Guardrails AI validates structured outputs (JSON Schema, function calling) with its validators.

The separation of responsibilities avoids overlap and allows pieces to be swapped without rewriting everything. All three expose a FastAPI API and emit OTel spans; the AI gateway orchestrates them sequentially.

Applied to on-premise hardware

On the RTX 4090 (24 GB)

A 4090 dedicated to the LLM Guard service pod comfortably serves the complete pipeline in medium-scale production:

• Anonymize (BERT-NER ONNX-INT8): ~50 MB VRAM.
• PromptInjection (DeBERTa ONNX-INT8): ~90 MB.
• Toxicity, Sentiment, Gibberish: ~150 MB total.
• BanTopics (BART-MNLI ONNX-INT8): ~200 MB.
• Bias, Relevance, FactualConsistency (output): ~250 MB total.

Total ~750 MB. The rest of the VRAM is idle or usable for aggressive batching. Sustained throughput of 3,000-6,000 RPS over the complete pipeline. For deployments with < 500 sustained RPS, the 4090 is under-used and can be shared with another workload (RAG embeddings, BGE reranker).

On the 4×H100 SXM cluster (320 GB total, NVLink)

There is capacity to spare by an order of magnitude. Sensible pattern:

• 3 H100s serving the main LLM at TP=3 (Llama 70B FP8).
• 1 H100 split into MIG instances (1g.10gb or similar) — one slice for LLM Guard (~10 GB MIG is more than enough), another for the reranker, another for embeddings.

Aggregate throughput for LLM Guard at that scale: 15,000-30,000 RPS. Plenty for a large multi-tenant setup with long sessions.

The specific operational pitfalls

Pitfall 1 — Vault with no TTL. The Vault grows without limit if it is not cleaned. In in-process lib mode per request there is no problem (the object is destroyed). In centralised service mode with Redis, the TTL is missing and Redis fills up. A silent trap discovered when the Redis pod gets OOM-killed in production after six weeks.

Pitfall 2 — Vault not shared between pods + AI gateway with no sticky session. If the AI gateway distributes round-robin across multiple LLM Guard pods, one pod’s local Vault knows nothing of the mapping created by another. Result: on turn 2 of a session, Deanonymize does not find the placeholders from turn 1 and leaves a literal [REDACTED_PERSON_1] in the answer. Solution: a shared Redis Vault or a sticky session by user_id.

Pitfall 3 — Models not exported to ONNX in production. It is deployed with the default config (Transformers) and latency is 3-5× worse than the benchmarks report. The team assumes LLM Guard “is slow”. The solution is to export to ONNX (built into the project) and configure recognizer_conf with the path to the model’s .onnx.

Pitfall 4 — fail_fast=False with many scanners. Without fail_fast, every scanner always runs, even if the first one already blocked. Latency is 3-5× worse on adversarial traffic. For production, barring an explicit reason (wanting complete per-scanner metrics even when blocking), fail_fast=True is the reasonable default.

Pitfall 5 — Infinite cache_ttl + prompts with variable PII. If the API’s cache stores the sanitized_prompt indefinitely, two different sessions with the same prompt structure but different PII can collide if the cache key does not include the Vault hash. You have to verify that the cache key includes either the complete content (without PII) or a hash of the original prompt.

Pitfall 6 — Structured logs with original PII. LLM Guard’s JSON stdout logs record only placeholders by default. But if custom hooks are added for debugging, it is easy to leak the original PII to the log. A regulatory audit (GDPR, ENS) detects this and it is a breach. Discipline: never add hooks that read from the Vault without explicit permission.

Pitfall 7 — scan_output without the original prompt. The scan_output method expects (prompt, output) for validators that compare both (Relevance, LanguageSame, FactualConsistency). If only the output is passed, those scanners fail silently or return is_valid=True by default. You have to keep the sanitized_prompt in the AI gateway and pass it to scan_output.

When to choose LLM Guard (and when not to)

Choose LLM Guard when:

• The requirement for PII anonymisation with automatic restitution is on the list. It is the number one reason to use it. Banking, healthcare, legal advice, HR — any case with heavy PII that must not reach the LLM even when it is local.
• You want a Pythonic pipeline with no new DSL. If the team is pure Python and prefers composing scanners as objects to learning Colang.
• The stack already has an AI gateway (LiteLLM, Envoy AI, Kong AI) and it integrates as a plugin without touching the app.
• You need built-in OTel and Prometheus with no additional instrumentation.

Do not choose LLM Guard when:

• The system is a conversational agent with complex dialogue flows (policies, fallbacks, escalation to a human). There NeMo Guardrails with Colang is structurally better.
• The safety layer reduces to validating structured outputs (JSON, function calling). Guardrails AI with its validators is more natural.
• Your latency budget is ultra-aggressive (< 30 ms for the whole layer). You will have to cut scanners and accept lower coverage; perhaps a single PromptGuard 2 + Presidio in a sidecar (the pattern from the guardrails post) is simpler.
• You do not want to carry the operational weight of a distributed Vault (Redis, TTL, sticky session). For small systems with no strong PII requirement, it is over-engineered.

What we have not covered (upcoming posts)

• Custom scanners: how to write your own scanner when nothing in the catalogue fits (a complex domain regex, your own fine-tuned classifier). The project supports custom scanners by inheriting from InputScanner / OutputScanner with three methods.
• Integration with SLSA / supply chain: how to sign the LLM Guard container with cosign, SLSA attestations, and verification in the cluster before admitting it. An operational supply chain security topic (OWASP LLM03).
• Red teaming against LLM Guard: known techniques that evade detectors (homoglyphs, Unicode confusables, base64 encoding inside the prompt). The project publishes an adversarial test suite for doing your own benchmarking. How it is set up as a continuous gate in CI.
• Comparative benchmark against Bedrock Guardrails and Azure AI Content Safety: F1 per category over real traffic across three different deployments. The OSS vs hyperscalers post has the strategic comparison; the technical detection comparison is still missing.

References

• LLM Guard (Protect AI): https://llm-guard.com — official documentation, scanner list, benchmarks.
• Repository: https://github.com/protectai/llm-guard.
• LLM Guard API: https://github.com/protectai/llm-guard/tree/main/llm_guard_api.
• Presidio (Microsoft): https://microsoft.github.io/presidio/ — basis of the Anonymize scanner.
• detect-secrets (Yelp): https://github.com/Yelp/detect-secrets — basis of the Secrets scanner.
• Langfuse OTel ingestion: https://langfuse.com/docs/opentelemetry/get-started.
• LiteLLM guardrails: https://docs.litellm.ai/docs/proxy/guardrails.
• Envoy AI Gateway: https://aigateway.envoyproxy.io.
• Kong AI Gateway: https://docs.konghq.com/hub/kong-inc/ai-prompt-guard/.
• OWASP Top 10 for LLM Applications 2025: https://owasp.org/www-project-top-10-for-large-language-model-applications/.
• ONNX Runtime: https://onnxruntime.ai — exporting HF models to ONNX for acceleration.

See also

• Hardening and secrets of the sovereign LLM stack: defence in depth — guardrails as one more layer of the stack’s defence in depth.

• Guardrails and safety in LLMs: the four lines of defence — the framework that places LLM Guard as one of the tools within the layer. That post explains the four lines (input, retrieval, tool, output), the OWASP LLM Top 10 and compares NeMo Guardrails, Llama Guard 4, ShieldGemma, Granite Guardian, PromptGuard 2 and LLM Guard from a bird’s-eye view.

• The OSS catalogue for LLMOps in six stages — an extended write-up of LLM Guard among the rest of the OSS tools by pipeline stage.

• RAG corpus curation: the active librarian — prevention at ingest shares the Presidio PII detector with LLM Guard; the Vault pattern is the new piece added at runtime.

• LLM tracing with OpenTelemetry GenAI — the OTel plane over which LLM Guard emits the gen_ai.guardrail.* spans that Langfuse and Tempo consume.

• Prompt versioning with Langfuse and MLflow — the prompt_id+version travels as a span attribute even when the prompt content is anonymised; it complements this post’s PII shielding.

• Evals for LLMs: the layer after tracing — LLM Guard’s offline counterpart. When a scanner reports a high FP rate over real traffic, the offline exercise against an annotated golden set identifies whether to tune the threshold or change the backend model.

• Retrain: closing the feedback → dataset → adapter loop — the HIGH severity incidents that LLM Guard emits with risk_score > threshold feed the incident-driven retrain loop.

• OSS vs hyperscalers in LLMOps — the OSS column of the “Guardrails” row (NeMo + Presidio + Llama Guard 4 + LLM Guard) against Bedrock Guardrails, Azure AI Content Safety and Vertex Model Armor.

• Structured output: function calling and constrained decoding — LLM Guard’s JSON scanner validates the output structure as a safety net when the inference engine has already done constrained decoding.

• The six-stage LLMOps pipeline — the master map where Guardrails (this post included) is the online counterpart of the Eval stage.