Structured output: the form with dropdowns that strikes out invalid answers before the model picks one — Outlines, XGrammar, LLGuidance and the maths of the bitmask

This post complements those on Continuous batching (where the scheduler lives) and Speculative decoding (another technique operating on the last mile of the sampler). Structured output is the LLM’s output contract towards the code that consumes it; without it, the integration between an LLM and applications is fragile by default.

You are here: DEPLOY

TL;DR

An LLM produces free text, but many applications (function calling, entity extraction, routing, text-to-SQL, config generation) need to parse it as JSON, as a tool call with typed args, as a valid SQL statement, or as one option from an enum. The naive solutions fail: prompt engineering (“answer in JSON”) leaves 25 % of outputs unparseable in many models; post-hoc validation plus retry costs latency and does not guarantee termination; json-repair libraries are heuristic patches. Constrained decoding guarantees conformance 100 %, by construction: at every generation step, before sampling from the softmax over the V vocabulary tokens, the tokens that would break the target structure are masked to -∞. The output is valid by mathematical contract, not by luck. The four dominant families in May 2026 are Outlines (Willard & Louf, 2023; FSM + token trie precomputed from regex/JSON Schema/CFG), XGrammar (Dong et al., 2024-25, CMU+MLC; byte-level pushdown automaton with an adaptive cache of context-independent tokens, the default in vLLM v1, SGLang, TensorRT-LLM, NIM and MLC-LLM), LLGuidance (Microsoft Research; Earley parser + regex derivatives, ~50 µs of CPU per token, underneath OpenAI Structured Outputs) and LM Format Enforcer (noamgat; aimed at JSON Schema, integrated in many engines as a fallback). XGrammar-2 (May 2026) introduces Structural Tag and a cross-grammar cache for dynamic agentic tool calling. The real cost when well integrated: <5 % in TPOT and ~40 µs of CPU per token of mask computation, partially overlappable with the model’s forward pass. The open operational question: does it degrade reasoning? The paper Let me Speak Freely? (Tam et al., EMNLP 2024) reported significant degradation in reasoning under format constraints; the dottxt rebuttal showed that the effect came from prompts that were not equivalent between experiments. The emerging consensus in May 2026: use two-pass (free reasoning with CoT in text plus structured output in a second call) for tasks that require multi-step reasoning; use single-pass constrained for extraction, classification and function calling, where forced structure improves exactness and reduces hallucinations. This post takes apart the mechanism, the maths (bitmask size = V/8 bytes, per-step latency), the comparison table of backends, the pitfalls (compile time, tokenizer-specific FSM, streaming SSE) and the production deployment pattern.

The analogy: the form with dropdowns instead of free-text fields

Imagine two ways of asking someone to fill in a form.

The first way is to hand them the paper with blank fields and say “fill it in with this exact format: the name goes here, then the ID number without spaces, then the date as YYYY-MM-DD, then mark one of these five possible reasons separated by semicolons”. You explain the format in exhaustive detail, but the person is still free to write whatever they like in each field. If they are in a hurry they may drop a zero from the ID number, put the date in the American format, tick two reasons when only one was asked for. When you receive the form, you often have to send it back: field X badly formatted, reason Y invalid, date Z impossible. This is exactly what an LLM does when you ask it to “answer in JSON with this schema”: it works most of the time, it fails a non-negligible percentage, and you have no formal guarantee of anything.

The second way is to give them an electronic form where the fields are not free text. The name accepts any text, but the ID number has a mask that only lets you type eight digits followed by a letter; the date is a picker that only lets you choose valid dates; the reasons are a dropdown with five closed options. The person can type whatever they want, but the form does not accept invalid characters at any point. The result is parseable by construction: when you receive the completed form, the ID number has exactly the expected format, the date is a real date, the reason is one of the five. This is constrained decoding.

The analogy holds on four mappings:

• The person filling it in = the LLM producing logits over the vocabulary.
• The characters the form lets you type in each field = the bitmask applied to the logits before sampling.
• How the form knows which characters to allow depending on which field you are in = the automaton (FSM or PDA) that holds the current state of the grammar.
• The fact that the dropdown is precomputed when the page loads rather than recomputed each time = the precomputed table of valid tokens per FSM state, which makes the per-step cost amortised O(1).

The problem structured output solves

The operational problem is the contract between the LLM and the code that consumes its output. There are three naive approaches, all with documented failures:

Pure prompt engineering. “Answer only with valid JSON, no comments or prose”. It works most of the time for good models; it fails between 5 % and 25 % of the time depending on the model, the temperature, the complexity of the schema and the length of the output. The model adds a trailing comma, escapes a quote badly, wraps the JSON in a ```json markdown block, hallucinates a field that was not in the schema, ignores a required field. SqueezeBits measures a ≤72 % correct rate without constraining for some models on JSON Schemas of moderate complexity.

Post-hoc validation plus retry. The server receives the output, tries to parse it, and if that fails returns the error to the model and asks it to try again. Cost: 2-3× latency in the worst case (typically 2-3 retries before giving up), no guarantee of termination, noise in the logs, hard to test deterministically.

JSON repair libraries (json_repair, the fast-json-repair Rust port). Heuristic patches for common errors: trailing commas, missing quotes, prose interleaved with JSON. Useful as a fallback for approach 2; they are not a contract.

The accumulated operational cost: latency inflated by retries, noise in production, painful debugging of intermittent parse errors, broken contracts with downstream clients that assumed clean parsing.

Constrained decoding: the principle

At each decode step, the model produces a vector logits ∈ R^V where V is the vocabulary size (Llama 3: 128,256, GPT-4o: ~200,000, Qwen 3: 152,064). The conventional sampler applies softmax plus a sampling strategy (greedy, top-k, top-p, temperature) over the V tokens.

Constrained decoding interposes one operation before the softmax: it applies a bitmask that sets to -∞ the logits of the tokens that would violate the target grammar at this step. The result: the softmax only assigns probability mass to admissible tokens; sampling, whatever its strategy, can only pick a valid one.

The two operational questions are always the same:

1. How do you know which tokens are valid at each step? → hold the current state of an automaton (FSM/PDA) built from the grammar; look up the table (state → set of valid tokens).
2. How much does it cost to compute and apply the mask? → this is where Outlines, XGrammar, LLGuidance and LM Format Enforcer compete.

The four families of backends

Three operational observations:

1. XGrammar is the de facto default in May 2026 across the open-source ecosystem (vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM). Its byte-level PDA with an adaptive cache gives it almost zero overhead when well integrated.
2. LLGuidance is the least known but most used piece on the market, because it sits underneath OpenAI Structured Outputs (confirmed in the README of the repo itself). 50 µs of CPU per token for 128k tokenizers.
3. Outlines was the first, it still holds the conceptual mindshare (papers, canonical blog posts), but it lost operational ground to XGrammar. The Rust version (outlines-core) closes part of the gap.

XGrammar in detail (what sits under vLLM and SGLang)

The paper by Dong, Yin, Ruan and Chen (arXiv:2411.15100, November 2024) introduces a partitioning technique that is key to understanding why XGrammar is 3-100× faster than the alternatives.

Partitioning the vocabulary at each PDA state:

• Context-independent tokens (~99 % of the vocab in a typical JSON Schema): their validity is decided only by the current PDA position, without needing to inspect the full stack. These are precomputable and stored as bitmasks in a cache.
• Context-dependent tokens (~1 %): they require inspecting the PDA stack at runtime. They are handled case by case at a higher cost.

That partition is the fundamental reason why XGrammar works in production at low TPOT. 99 % of the lookups go to a precomputed table in O(1); only the residual 1 % pays the real cost.

Other combined techniques: a pushdown automaton for full CFGs (not just regex), a persistent stack for fast branching/rollback, JIT compilation plus an Earley parser in XGrammar-2.

Reported speedups (paper):

• Up to 100× over previous solutions on CFG.
• 3× on JSON Schema versus Outlines.
• Latency <40 µs per token on JSON Schema; <200 µs on XML/Python DSL.
• When well integrated in vLLM/SGLang/TRT-LLM: near-zero end-to-end overhead.

XGrammar-2 (May 2026, arXiv:2601.04426) adds two pieces that matter for agents:

• Structural Tag: a composable JSON protocol that unifies OpenAI harmony, tool calling and reasoning channels.
• Cross-Grammar Cache for reuse at sub-structure level → it allows dynamic switching between sub-grammars in agentic loops without recompiling.
• 6× faster compile time than XGrammar-1.

The maths of the bitmask

Three numbers drive the operational decision.

Bitmask size per step. If the vocabulary has V tokens, the bit-packed bitmask takes V/8 bytes:

• Llama 3 (V=128,256): 16 KB per bitmask per request per step.
• GPT-4o tokenizer o200k (V≈200,000): 25 KB.
• Llama 2 (V=32,000): 4 KB.

For a batch of 32 requests with structured output active on an H100, we are talking about ~512 KB of bitmasks per step, trivial against the GBs the forward pass moves.

Cost of applying the bitmask to the logits. A simple CUDA kernel, O(V) complexity, ~5-10 µs of latency on an H100. Negligible.

Cost of computing the bitmask (CPU-side). This is where backends differ:

With a model forward pass on the order of 10-50 ms per token in decode, a 40-50 µs mask is <0.5 % overhead, invisible. The operational key is that the CPU-side mask computation can be overlapped with the GPU-side forward pass: while the GPU computes the logits for token t, the CPU precomputes the mask for token t+1 based on the FSM state after token t-1. SGLang does this explicitly; vLLM v1 improved substantially over v0.

Compile-time cost. Precomputing the FSM/PDA and the cache:

• Simple schemas (1-5 fields): <100 ms.
• Deep JSON Schemas with many $defs: seconds.
• OpenAI structured outputs: “10s typical, up to 1 minute for complex schemas”, cached after the first call.
• XGrammar-2: 6× faster than XGrammar-1.
• LLGuidance: ~2 ms startup.

Operational best practice: pre-cache the known schemas when the server starts, to avoid latency spikes on the first request for each new schema.

Does it degrade the model’s reasoning?

It is the most interesting question in the field and the answer is not obvious.

The theoretical argument for degradation: forcing the structure changes the model’s probability distribution; if the structured path pushes towards low-probability paths, the model gets “stuck” in a suboptimal branch with no room to explore.

The paper Let me Speak Freely? (Tam et al., arXiv:2408.02442, EMNLP 2024 Industry Track) reported significant degradation on reasoning tasks under format constraints (JSON/XML/YAML). The stricter the format, the greater the degradation. The paper itself, however, acknowledged an improvement on classification tasks with forced structure.

The dottxt rebuttal, Say What You Mean (official blog post): a methodological critique. The prompts in the original paper were different between structured and unstructured (not apples to apples). The JSON prompts in the original experiment gave less information than the unstructured ones. Re-running with equivalent prompts (Llama-3-8B-Instruct), dottxt does not reproduce the degradation. The conclusion: the paper confused the format constraint with the prompt engineering of that constraint.

Emerging consensus in May 2026 (gathered from technical blogs and community empiricism):

• For extraction, classification, function calling and routing: constrained decoding improves exactness and reduces hallucinations. It is the right tool.
• For multi-step reasoning (maths, logic, code review): use two-pass:
  1. First call: free reasoning with Chain-of-Thought in natural text (generous max_tokens, no constraint).
  2. Second call: pass it the reasoning plus the schema, and ask it to produce structured output with constrained decoding.

The “reason then structure” pattern gives the best of both worlds: reasoning without a straitjacket plus an output guaranteed to be parseable.

Real implementations in May 2026

vLLM example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[{"role": "user", "content": "Extract name and age from: María is 34 years old"}],
    extra_body={
        "guided_json": {
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "age": {"type": "integer"}
            },
            "required": ["name", "age"]
        }
    }
)

SGLang equivalent:

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[...],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person",
            "strict": True,
            "schema": {...}
        }
    }
)

Usage patterns in production

Function calling / tool use. The LLM produces {tool_name: enum, arguments: object} according to a schema. The dominant case today (OpenAI, Anthropic, open models). The schema guarantees that tool_name is in the set of available tools and that arguments has the right types.

Entity extraction. A schema with {name, address, phone, ...} from unstructured text. It moves schema adherence from 76% to 98% in vendor benchmarks.

Routing / classification. The LLM picks between N options (enum in the schema). The bitmask shrinks to a handful of valid tokens → almost zero overhead, maximum reliability.

Constrained SQL. An SQL grammar as GBNF or Lark → it avoids injection and syntax errors. Useful in text-to-SQL agents.

Code generation with a valid AST. A grammar for the target language (Python, Rust, a custom DSL). It guarantees that the output is compilable or parseable code.

Agentic loops. XGrammar-2 Structural Tag for dynamic switching between formats (tool call → reasoning channel → tool result) without recompiling the grammar.

Operational pitfalls

Compile time for large schemas. JSON Schemas with many $defs can take seconds to compile. This causes slow startup if they are loaded at boot, or latency spikes on the first request if they are loaded on demand. Mitigation: warm the cache at boot with the fleet’s known schemas.

Tokenizer-specific FSM/PDA. A schema precompiled for Llama 3 (128k tokenizer) does not work for Qwen (152k). The schema cache must be keyed by (tokenizer_hash, schema_hash). Changing model = invalidating the cache.

Schema changes. Version schemas explicitly. Breaking changes → rebuild plus warm cache. Do not silence compile errors in production.

Streaming SSE. Structured output with streaming requires parsers tolerant of partial output (partial-json-parser, json-stream). Strict Pydantic v1 fails; v2 with partial validation works. Older clients may not handle incremental validation, so test with the real client.

Token healing. Tokens that cross boundaries (:// as a single token versus : + //) can break pure constraints. Outlines and XGrammar mitigate this internally; llama.cpp and others require explicit token healing. If the model sneaks a “strange” character in after the structure, this is probably why.

Real known bugs in vLLM 2025-26:

• vLLM 0.8.4: XGrammar rejects minItems in JSON Schema (issue #16880).
• vLLM with Qwen 2.5 VL: Outlines/XGrammar does not respect the schema in some cases (issue #13038).
• The vLLM apply_token_bitmask_inplace bitmask backend is always auto, not exposed to the user.

OpenAI subset limitations: if you are going to port a schema validated on OpenAI structured outputs to another backend, check that you are not using constructs that OpenAI rejects but others allow (deep $ref, pattern, default, depth >5, anyOf as root).

Implications on on-premise hardware

On an RTX 4090 (24 GB). Any model you serve with vLLM or llama.cpp can carry structured output at practically negligible cost. The CPU-side mask latency (~40 µs) is trivial compared with typical TPOT (30-100 ms on consumer hardware). The interesting case: function calling over Llama 3 8B or Qwen 3 14B INT4 → reliable tool use without retries, with no need for a hosted API.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Here XGrammar as the vLLM v1 / SGLang default is the way:

• Llama 3 70B FP8 + XGrammar JSON Schema: TPOT P95 stable under 60 ms even with structured output active across the whole batch. It supports hundreds of distinct cached schemas.
• DeepSeek-V3 + XGrammar Structural Tag: agentic tool calling with native MTP and constrained decoding combined; the mask cost overlaps with the MoE forward pass.
• Multi-tenant function calling: each client can have its own set of tools (that is, its own schemas); compile time is amortised through caching, runtime is invariant.

The rule of thumb for May 2026: XGrammar by default, pre-cache the fleet’s schemas at boot, two-pass for reasoning tasks.

What we have not covered

• Dynamic tool routing with XGrammar-2 Structural Tag: the detail of how TagDispatch picks sub-grammars at runtime.
• Constrained beam search and its interaction with grammar: theoretical quality degradation versus greedy.
• Grammars for code generation with a full AST: production-grade Python/Rust grammars, performance.
• JSON Schema → Pydantic → grammar pipelines: tooling to reduce human error.
• Inhibition decoding (the Inhibition Decoding paper, 2025): a variant that penalises but does not forbid certain tokens, useful for soft safety constraints.

See also

• Continuous batching: the hairdresser with 8 chairs — the scheduler where structured output is applied request by request; the CPU-side mask computation can overlap with the GPU-side forward pass.
• Speculative decoding — another technique operating on the sampler; speculative plus structured can be combined but requires care (the acceptance rule has to respect the bitmask).
• Multi-LoRA serving — an adapter may be trained specifically for function calling, complementing the structured output guarantee with the model’s affinity for the task.
• LLM-as-judge: the exam marker — the judge produces a structured verdict ({score, reasoning, decision}) that can be secured with structured output to avoid manual parsing.
• Evals for LLMs — evals with LLM-as-judge benefit enormously from guaranteed structured output.
• The six-stage LLMOps pipeline — the master map where Deploy is stage 4.
• Ontologies and knowledge graphs in LLMOps — the JSON Schemas imposed on the sampler here usually derive from a SHACL shape of the corporate ontology; structured output is the mechanism by which the LLM populates the KG’s ABox in line with the declared TBox.
• Function calling and tool-augmented retrieval: the detective who knows which file to ask for — the JSON Schema defining each tool call is structured output applied to the tool interface; the constrained decoding guarantee from this post is what makes the LLM generate calls that parse 100 % of the time.

References

• Willard, B., Louf, R. Efficient Guided Generation for Large Language Models (Outlines). 2023. https://arxiv.org/abs/2307.09702
• Dong, Y., Yin, X., Ruan, F., Chen, T. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. 2024. https://arxiv.org/abs/2411.15100
• XGrammar-2: Dynamic Structured Generation for Agentic LLMs. 2026. https://arxiv.org/abs/2601.04426
• Tam, Z. et al. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP 2024 Industry. https://arxiv.org/abs/2408.02442
• dottxt blog, Say What You Mean: https://blog.dottxt.ai/say-what-you-mean.html
• OpenAI, Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/
• Outlines repo: https://github.com/dottxt-ai/outlines
• outlines-core (Rust): https://github.com/dottxt-ai/outlines-core
• XGrammar repo: https://github.com/mlc-ai/xgrammar
• LLGuidance (Microsoft Research) repo: https://github.com/guidance-ai/llguidance
• LM Format Enforcer repo: https://github.com/noamgat/lm-format-enforcer
• llama.cpp grammars: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md
• vLLM Structured Outputs docs: https://docs.vllm.ai/en/stable/features/structured_outputs/
• SGLang Structured Outputs docs: https://docs.sglang.io/advanced_features/structured_outputs.html
• TensorRT-LLM guided decoding (Triton): https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/docs/guided_decoding.html
• NIM Structured Generation: https://docs.nvidia.com/nim/large-language-models/1.12.0/structured-generation.html
• MLC blog, XGrammar: https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar
• MLC blog, XGrammar-2: https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation
• SqueezeBits, Guided decoding performance: vLLM vs SGLang: https://blog.squeezebits.com/guided-decoding-performance-vllm-sglang
• Red Hat, Structured outputs in vLLM: https://developers.redhat.com/articles/2025/06/03/structured-outputs-vllm-guiding-ai-responses
• Aidan Cooper, Constrained Decoding guide: https://www.aidancooper.co.uk/constrained-decoding/