Function-Calling on lo0 — Tech Blog

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

Structured output: the form with dropdowns that strikes out invalid answers before the model picks one — Outlines, XGrammar, LLGuidance and the maths of the bitmask

Sat, 30 May 2026 16:30:00 +0200

This post complements those on Continuous batching (where the scheduler lives) and Speculative decoding (another technique operating on the last mile of the sampler). Structured output is the LLM’s output contract towards the code that consumes it; without it, the integration between an LLM and applications is fragile by default.

You are here: DEPLOY

TL;DR

An LLM produces free text, but many applications (function calling, entity extraction, routing, text-to-SQL, config generation) need to parse it as JSON, as a tool call with typed args, as a valid SQL statement, or as one option from an enum. The naive solutions fail: prompt engineering (“answer in JSON”) leaves 25 % of outputs unparseable in many models; post-hoc validation plus retry costs latency and does not guarantee termination; json-repair libraries are heuristic patches. Constrained decoding guarantees conformance 100 %, by construction: at every generation step, before sampling from the softmax over the V vocabulary tokens, the tokens that would break the target structure are masked to -∞. The output is valid by mathematical contract, not by luck. The four dominant families in May 2026 are Outlines (Willard & Louf, 2023; FSM + token trie precomputed from regex/JSON Schema/CFG), XGrammar (Dong et al., 2024-25, CMU+MLC; byte-level pushdown automaton with an adaptive cache of context-independent tokens, the default in vLLM v1, SGLang, TensorRT-LLM, NIM and MLC-LLM), LLGuidance (Microsoft Research; Earley parser + regex derivatives, ~50 µs of CPU per token, underneath OpenAI Structured Outputs) and LM Format Enforcer (noamgat; aimed at JSON Schema, integrated in many engines as a fallback). XGrammar-2 (May 2026) introduces Structural Tag and a cross-grammar cache for dynamic agentic tool calling. The real cost when well integrated: <5 % in TPOT and ~40 µs of CPU per token of mask computation, partially overlappable with the model’s forward pass. The open operational question: does it degrade reasoning? The paper Let me Speak Freely? (Tam et al., EMNLP 2024) reported significant degradation in reasoning under format constraints; the dottxt rebuttal showed that the effect came from prompts that were not equivalent between experiments. The emerging consensus in May 2026: use two-pass (free reasoning with CoT in text plus structured output in a second call) for tasks that require multi-step reasoning; use single-pass constrained for extraction, classification and function calling, where forced structure improves exactness and reduces hallucinations. This post takes apart the mechanism, the maths (bitmask size = V/8 bytes, per-step latency), the comparison table of backends, the pitfalls (compile time, tokenizer-specific FSM, streaming SSE) and the production deployment pattern.

The analogy: the form with dropdowns instead of free-text fields

Imagine two ways of asking someone to fill in a form.

The first way is to hand them the paper with blank fields and say “fill it in with this exact format: the name goes here, then the ID number without spaces, then the date as YYYY-MM-DD, then mark one of these five possible reasons separated by semicolons”. You explain the format in exhaustive detail, but the person is still free to write whatever they like in each field. If they are in a hurry they may drop a zero from the ID number, put the date in the American format, tick two reasons when only one was asked for. When you receive the form, you often have to send it back: field X badly formatted, reason Y invalid, date Z impossible. This is exactly what an LLM does when you ask it to “answer in JSON with this schema”: it works most of the time, it fails a non-negligible percentage, and you have no formal guarantee of anything.

The second way is to give them an electronic form where the fields are not free text. The name accepts any text, but the ID number has a mask that only lets you type eight digits followed by a letter; the date is a picker that only lets you choose valid dates; the reasons are a dropdown with five closed options. The person can type whatever they want, but the form does not accept invalid characters at any point. The result is parseable by construction: when you receive the completed form, the ID number has exactly the expected format, the date is a real date, the reason is one of the five. This is constrained decoding.

The analogy holds on four mappings:

The person filling it in = the LLM producing logits over the vocabulary.
The characters the form lets you type in each field = the bitmask applied to the logits before sampling.
How the form knows which characters to allow depending on which field you are in = the automaton (FSM or PDA) that holds the current state of the grammar.
The fact that the dropdown is precomputed when the page loads rather than recomputed each time = the precomputed table of valid tokens per FSM state, which makes the per-step cost amortised O(1).

The problem structured output solves

The operational problem is the contract between the LLM and the code that consumes its output. There are three naive approaches, all with documented failures:

Pure prompt engineering. “Answer only with valid JSON, no comments or prose”. It works most of the time for good models; it fails between 5 % and 25 % of the time depending on the model, the temperature, the complexity of the schema and the length of the output. The model adds a trailing comma, escapes a quote badly, wraps the JSON in a ```json markdown block, hallucinates a field that was not in the schema, ignores a required field. SqueezeBits measures a ≤72 % correct rate without constraining for some models on JSON Schemas of moderate complexity.

Post-hoc validation plus retry. The server receives the output, tries to parse it, and if that fails returns the error to the model and asks it to try again. Cost: 2-3× latency in the worst case (typically 2-3 retries before giving up), no guarantee of termination, noise in the logs, hard to test deterministically.

JSON repair libraries (json_repair, the fast-json-repair Rust port). Heuristic patches for common errors: trailing commas, missing quotes, prose interleaved with JSON. Useful as a fallback for approach 2; they are not a contract.

The accumulated operational cost: latency inflated by retries, noise in production, painful debugging of intermittent parse errors, broken contracts with downstream clients that assumed clean parsing.

Constrained decoding: the principle

At each decode step, the model produces a vector logits ∈ R^V where V is the vocabulary size (Llama 3: 128,256, GPT-4o: ~200,000, Qwen 3: 152,064). The conventional sampler applies softmax plus a sampling strategy (greedy, top-k, top-p, temperature) over the V tokens.

Constrained decoding interposes one operation before the softmax: it applies a bitmask that sets to -∞ the logits of the tokens that would violate the target grammar at this step. The result: the softmax only assigns probability mass to admissible tokens; sampling, whatever its strategy, can only pick a valid one.

The two operational questions are always the same:

How do you know which tokens are valid at each step? → hold the current state of an automaton (FSM/PDA) built from the grammar; look up the table (state → set of valid tokens).
How much does it cost to compute and apply the mask? → this is where Outlines, XGrammar, LLGuidance and LM Format Enforcer compete.

The four families of backends

Backend	Origin	Algorithm	Grammar formats	Default in	Notes
Outlines	dottxt (2023)	FSM + precomputed token trie	regex, JSON Schema, Lark CFG	HF TGI	The Rust version (outlines-core) closes part of the gap with XGrammar
XGrammar	CMU + MLC (2024-25)	Byte-level PDA + adaptive ctx-indep/dep cache	regex, JSON Schema, EBNF/CFG	vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM	Speedup vs naive: up to 100× CFG, 3× JSON; <40 µs/token JSON
LLGuidance	Microsoft Research	Earley parser + regex derivatives	regex, JSON Schema, Lark	OpenAI Structured Outputs (internally), Chromium	~50 µs CPU/token; underneath Guidance, llama.cpp, mistral.rs
LM Format Enforcer	noamgat	char-level parser + tokenizer prefix tree	JSON Schema, regex	(deprecated default in many engines, fallback in NIM)	Slower than XGrammar (3.5× worse JSON, 10× worse CFG)

Three operational observations:

XGrammar is the de facto default in May 2026 across the open-source ecosystem (vLLM v1, SGLang, TensorRT-LLM, NIM, MLC-LLM). Its byte-level PDA with an adaptive cache gives it almost zero overhead when well integrated.
LLGuidance is the least known but most used piece on the market, because it sits underneath OpenAI Structured Outputs (confirmed in the README of the repo itself). 50 µs of CPU per token for 128k tokenizers.
Outlines was the first, it still holds the conceptual mindshare (papers, canonical blog posts), but it lost operational ground to XGrammar. The Rust version (outlines-core) closes part of the gap.

XGrammar in detail (what sits under vLLM and SGLang)

The paper by Dong, Yin, Ruan and Chen (arXiv:2411.15100, November 2024) introduces a partitioning technique that is key to understanding why XGrammar is 3-100× faster than the alternatives.

Partitioning the vocabulary at each PDA state:

Context-independent tokens (~99 % of the vocab in a typical JSON Schema): their validity is decided only by the current PDA position, without needing to inspect the full stack. These are precomputable and stored as bitmasks in a cache.
Context-dependent tokens (~1 %): they require inspecting the PDA stack at runtime. They are handled case by case at a higher cost.

That partition is the fundamental reason why XGrammar works in production at low TPOT. 99 % of the lookups go to a precomputed table in O(1); only the residual 1 % pays the real cost.

Other combined techniques: a pushdown automaton for full CFGs (not just regex), a persistent stack for fast branching/rollback, JIT compilation plus an Earley parser in XGrammar-2.

Reported speedups (paper):

Up to 100× over previous solutions on CFG.
3× on JSON Schema versus Outlines.
Latency <40 µs per token on JSON Schema; <200 µs on XML/Python DSL.
When well integrated in vLLM/SGLang/TRT-LLM: near-zero end-to-end overhead.

XGrammar-2 (May 2026, arXiv:2601.04426) adds two pieces that matter for agents:

Structural Tag: a composable JSON protocol that unifies OpenAI harmony, tool calling and reasoning channels.
Cross-Grammar Cache for reuse at sub-structure level → it allows dynamic switching between sub-grammars in agentic loops without recompiling.
6× faster compile time than XGrammar-1.

The maths of the bitmask

Three numbers drive the operational decision.

Bitmask size per step. If the vocabulary has V tokens, the bit-packed bitmask takes V/8 bytes:

$$\text{bitmask size} = \frac{V}{8} \text{ bytes}$$

Llama 3 (V=128,256): 16 KB per bitmask per request per step.
GPT-4o tokenizer o200k (V≈200,000): 25 KB.
Llama 2 (V=32,000): 4 KB.

For a batch of 32 requests with structured output active on an H100, we are talking about ~512 KB of bitmasks per step, trivial against the GBs the forward pass moves.

Cost of applying the bitmask to the logits. A simple CUDA kernel, O(V) complexity, ~5-10 µs of latency on an H100. Negligible.

Cost of computing the bitmask (CPU-side). This is where backends differ:

Backend	CPU latency per token (Llama 3, V=128k)
LLGuidance	~50 µs
XGrammar	~40 µs (JSON Schema), ~200 µs (XML/DSL)
outlines-core	comparable to XGrammar after 2024
Outlines Python (legacy)	200-1000 µs
LM Format Enforcer	intermediate, degrades with a large vocab

With a model forward pass on the order of 10-50 ms per token in decode, a 40-50 µs mask is <0.5 % overhead, invisible. The operational key is that the CPU-side mask computation can be overlapped with the GPU-side forward pass: while the GPU computes the logits for token t, the CPU precomputes the mask for token t+1 based on the FSM state after token t-1. SGLang does this explicitly; vLLM v1 improved substantially over v0.

Compile-time cost. Precomputing the FSM/PDA and the cache:

Simple schemas (1-5 fields): <100 ms.
Deep JSON Schemas with many $defs: seconds.
OpenAI structured outputs: “10s typical, up to 1 minute for complex schemas”, cached after the first call.
XGrammar-2: 6× faster than XGrammar-1.
LLGuidance: ~2 ms startup.

Operational best practice: pre-cache the known schemas when the server starts, to avoid latency spikes on the first request for each new schema.

Does it degrade the model’s reasoning?

It is the most interesting question in the field and the answer is not obvious.

The theoretical argument for degradation: forcing the structure changes the model’s probability distribution; if the structured path pushes towards low-probability paths, the model gets “stuck” in a suboptimal branch with no room to explore.

The paper Let me Speak Freely? (Tam et al., arXiv:2408.02442, EMNLP 2024 Industry Track) reported significant degradation on reasoning tasks under format constraints (JSON/XML/YAML). The stricter the format, the greater the degradation. The paper itself, however, acknowledged an improvement on classification tasks with forced structure.

The dottxt rebuttal, Say What You Mean (official blog post): a methodological critique. The prompts in the original paper were different between structured and unstructured (not apples to apples). The JSON prompts in the original experiment gave less information than the unstructured ones. Re-running with equivalent prompts (Llama-3-8B-Instruct), dottxt does not reproduce the degradation. The conclusion: the paper confused the format constraint with the prompt engineering of that constraint.

Emerging consensus in May 2026 (gathered from technical blogs and community empiricism):

For extraction, classification, function calling and routing: constrained decoding improves exactness and reduces hallucinations. It is the right tool.
For multi-step reasoning (maths, logic, code review): use two-pass:
1. First call: free reasoning with Chain-of-Thought in natural text (generous max_tokens, no constraint).
2. Second call: pass it the reasoning plus the schema, and ask it to produce structured output with constrained decoding.

The “reason then structure” pattern gives the best of both worlds: reasoning without a straitjacket plus an output guaranteed to be parseable.

Real implementations in May 2026

Engine	Default backend	Others available	API
vLLM v1	XGrammar (auto)	Outlines, Guidance (llguidance), LM Format Enforcer	`guided_json`, `guided_regex`, `guided_choice`, `guided_grammar` in the request
SGLang	XGrammar	Outlines, LLGuidance	`response_format.json_schema`, `extra_body.regex`, `extra_body.ebnf`
TensorRT-LLM	XGrammar	LLGTRT (Rust llguidance)	Official integration since Jan 2025
NVIDIA NIM	XGrammar (switched from Outlines in 2025)	LM Format Enforcer (requires `NIM_ENABLE_KV_CACHE_REUSE=0`)	Pluggable multi-backend
llama.cpp	Native GBNF	LLGuidance	Each candidate token is tested against the parse state
HF TGI	Outlines	XGrammar (experimental)	“Guidance” feature with `/generate` and `/chat/completion` with `tools`
MLC-LLM	XGrammar (native, same team)	—	Its own API

vLLM example:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-70B-Instruct",
 messages=[{"role": "user", "content": "Extract name and age from: María is 34 years old"}],
 extra_body={
 "guided_json": {
 "type": "object",
 "properties": {
 "name": {"type": "string"},
 "age": {"type": "integer"}
 },
 "required": ["name", "age"]
 }
 }
)

SGLang equivalent:

response = client.chat.completions.create(
 model="meta-llama/Llama-3.1-70B-Instruct",
 messages=[...],
 response_format={
 "type": "json_schema",
 "json_schema": {
 "name": "person",
 "strict": True,
 "schema": {...}
 }
 }
)

Usage patterns in production

Function calling / tool use. The LLM produces {tool_name: enum, arguments: object} according to a schema. The dominant case today (OpenAI, Anthropic, open models). The schema guarantees that tool_name is in the set of available tools and that arguments has the right types.

Entity extraction. A schema with {name, address, phone, ...} from unstructured text. It moves schema adherence from 76% to 98% in vendor benchmarks.

Routing / classification. The LLM picks between N options (enum in the schema). The bitmask shrinks to a handful of valid tokens → almost zero overhead, maximum reliability.

Constrained SQL. An SQL grammar as GBNF or Lark → it avoids injection and syntax errors. Useful in text-to-SQL agents.

Code generation with a valid AST. A grammar for the target language (Python, Rust, a custom DSL). It guarantees that the output is compilable or parseable code.

Agentic loops. XGrammar-2 Structural Tag for dynamic switching between formats (tool call → reasoning channel → tool result) without recompiling the grammar.

Operational pitfalls

Compile time for large schemas. JSON Schemas with many $defs can take seconds to compile. This causes slow startup if they are loaded at boot, or latency spikes on the first request if they are loaded on demand. Mitigation: warm the cache at boot with the fleet’s known schemas.

Tokenizer-specific FSM/PDA. A schema precompiled for Llama 3 (128k tokenizer) does not work for Qwen (152k). The schema cache must be keyed by (tokenizer_hash, schema_hash). Changing model = invalidating the cache.

Schema changes. Version schemas explicitly. Breaking changes → rebuild plus warm cache. Do not silence compile errors in production.

Streaming SSE. Structured output with streaming requires parsers tolerant of partial output (partial-json-parser, json-stream). Strict Pydantic v1 fails; v2 with partial validation works. Older clients may not handle incremental validation, so test with the real client.

Token healing. Tokens that cross boundaries (:// as a single token versus : + //) can break pure constraints. Outlines and XGrammar mitigate this internally; llama.cpp and others require explicit token healing. If the model sneaks a “strange” character in after the structure, this is probably why.

Real known bugs in vLLM 2025-26:

vLLM 0.8.4: XGrammar rejects minItems in JSON Schema (issue #16880).
vLLM with Qwen 2.5 VL: Outlines/XGrammar does not respect the schema in some cases (issue #13038).
The vLLM apply_token_bitmask_inplace bitmask backend is always auto, not exposed to the user.

OpenAI subset limitations: if you are going to port a schema validated on OpenAI structured outputs to another backend, check that you are not using constructs that OpenAI rejects but others allow (deep $ref, pattern, default, depth >5, anyOf as root).

Implications on on-premise hardware

On an RTX 4090 (24 GB). Any model you serve with vLLM or llama.cpp can carry structured output at practically negligible cost. The CPU-side mask latency (~40 µs) is trivial compared with typical TPOT (30-100 ms on consumer hardware). The interesting case: function calling over Llama 3 8B or Qwen 3 14B INT4 → reliable tool use without retries, with no need for a hosted API.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Here XGrammar as the vLLM v1 / SGLang default is the way:

Llama 3 70B FP8 + XGrammar JSON Schema: TPOT P95 stable under 60 ms even with structured output active across the whole batch. It supports hundreds of distinct cached schemas.
DeepSeek-V3 + XGrammar Structural Tag: agentic tool calling with native MTP and constrained decoding combined; the mask cost overlaps with the MoE forward pass.
Multi-tenant function calling: each client can have its own set of tools (that is, its own schemas); compile time is amortised through caching, runtime is invariant.

The rule of thumb for May 2026: XGrammar by default, pre-cache the fleet’s schemas at boot, two-pass for reasoning tasks.

What we have not covered

Dynamic tool routing with XGrammar-2 Structural Tag: the detail of how TagDispatch picks sub-grammars at runtime.
Constrained beam search and its interaction with grammar: theoretical quality degradation versus greedy.
Grammars for code generation with a full AST: production-grade Python/Rust grammars, performance.
JSON Schema → Pydantic → grammar pipelines: tooling to reduce human error.
Inhibition decoding (the Inhibition Decoding paper, 2025): a variant that penalises but does not forbid certain tokens, useful for soft safety constraints.

References

Willard, B., Louf, R. Efficient Guided Generation for Large Language Models (Outlines). 2023. https://arxiv.org/abs/2307.09702
Dong, Y., Yin, X., Ruan, F., Chen, T. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. 2024. https://arxiv.org/abs/2411.15100
XGrammar-2: Dynamic Structured Generation for Agentic LLMs. 2026. https://arxiv.org/abs/2601.04426
Tam, Z. et al. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. EMNLP 2024 Industry. https://arxiv.org/abs/2408.02442
dottxt blog, Say What You Mean: https://blog.dottxt.ai/say-what-you-mean.html
OpenAI, Introducing Structured Outputs in the API: https://openai.com/index/introducing-structured-outputs-in-the-api/
Outlines repo: https://github.com/dottxt-ai/outlines
outlines-core (Rust): https://github.com/dottxt-ai/outlines-core
XGrammar repo: https://github.com/mlc-ai/xgrammar
LLGuidance (Microsoft Research) repo: https://github.com/guidance-ai/llguidance
LM Format Enforcer repo: https://github.com/noamgat/lm-format-enforcer
llama.cpp grammars: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md
vLLM Structured Outputs docs: https://docs.vllm.ai/en/stable/features/structured_outputs/
SGLang Structured Outputs docs: https://docs.sglang.io/advanced_features/structured_outputs.html
TensorRT-LLM guided decoding (Triton): https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/docs/guided_decoding.html
NIM Structured Generation: https://docs.nvidia.com/nim/large-language-models/1.12.0/structured-generation.html
MLC blog, XGrammar: https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar
MLC blog, XGrammar-2: https://blog.mlc.ai/2026/05/04/xgrammar-2-fast-customizable-structured-generation
SqueezeBits, Guided decoding performance: vLLM vs SGLang: https://blog.squeezebits.com/guided-decoding-performance-vllm-sglang
Red Hat, Structured outputs in vLLM: https://developers.redhat.com/articles/2025/06/03/structured-outputs-vllm-guiding-ai-responses
Aidan Cooper, Constrained Decoding guide: https://www.aidancooper.co.uk/constrained-decoding/

Function-Calling on lo0 — Tech Blog

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

TL;DR

The analogy: the detective who knows which file to ask for

What function calling is: the anatomy of a tool call

Defining tools in the system prompt

The life cycle of a tool call

Support in OSS models

The ReAct pattern: Reasoning + Acting

A full ReAct trace: a real production query

The three sources and when to use each one

Tool routing: how the LLM picks the right tool

Descriptions that work versus those that do not

Parallel tool calling

Tool selection accuracy: the eval metric

Error handling in tool calls

SQL injection via prompt

Rate limits, timeouts and tool errors

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

Total latency by number of iterations

Comparison with a naive RAG

On-premise hardware for ReAct agents

What we have not covered

See also

References

Structured output: the form with dropdowns that strikes out invalid answers before the model picks one — Outlines, XGrammar, LLGuidance and the maths of the bitmask

You are here: DEPLOY

TL;DR

The analogy: the form with dropdowns instead of free-text fields

The problem structured output solves

Constrained decoding: the principle

The four families of backends

XGrammar in detail (what sits under vLLM and SGLang)

The maths of the bitmask

Does it degrade the model’s reasoning?

Real implementations in May 2026

Usage patterns in production

Operational pitfalls

Implications on on-premise hardware

What we have not covered

See also

References