Serving reasoning models: the invisible scratchpad that decides your latency and your bill

Fri, 12 Jun 2026 05:30:00 +0200

TL;DR

A reasoning model generates, before the answer you see, a thinking block, hundreds or thousands of tokens enclosed between <think> and </think>, that the user does not read but that consumes the same compute, the same VRAM and the same bill as any other token. Serving them in production is not “serving a bigger model”: it is serving a model whose cost per request is variable and, by default, uncontrolled. Three levers change the equation: choosing well between think / non-think mode (the hybrid models of 2026 allow it within the same weights), setting a reasoning budget (thinking_token_budget in vLLM, s1-style budget forcing) and measuring the impact on the KV cache (a 30k-token reasoning chain can eat ~9 GB of KV in FP16). Managed well, you gain accuracy where it matters; managed badly, you pay up to 113× more energy for an answer that did not need that much thinking.

The analogy

Picture a candidate sitting a competitive exam. Before writing the good answer on the official sheet, they fill three sheets of scratch paper: they try approaches, correct themselves, discard. Those scratch sheets are not handed in, the board only reads the final answer, but the candidate has spent ink, time and half an hour of the exam on them.

A reasoning model does exactly that. The <think>…</think> block is the scratch paper: it reasons out loud, contradicts itself, backtracks, and finally emits the “clean” answer. The production problem is that you pay for the scratch paper too: every reasoning token takes a slot in the batch, grows the KV cache and adds to the bill just like an output token. And, like any nervous candidate, the model tends to overthink: it writes five sheets when one would have done.

The operational question in this post is not “does it reason well?”, but “how much scratch paper do I let it write, and when do I take the pen away?”.

What changes when the model “thinks”

In a normal model, a request is: prompt → output tokens. In a reasoning model the sequence has two stretches:

Reasoning block (<think> … </think>): the chain of thought (CoT). Long, variable, normally hidden from the end user.
Final answer: what the user sees.

vLLM models this explicitly. When started with a reasoning parser, the output carries a reasoning_content field separate from content:

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
 --reasoning-parser deepseek_r1

The parser detects the thinking delimiters and separates both stretches in the API response (vLLM · Reasoning Outputs). This matters for serving for one practical reason: if your gateway or your UI does not separate reasoning_content from content, you will end up showing the scratch paper, or, worse, recording it in logs and traces without meaning to.

The 2026 landscape: from the model that “always reasons” to the switchable hybrid

The first generation (o1, DeepSeek-R1, QwQ-32B) were models that always reasoned. In 2026 the dominant pattern is the hybrid: a single set of weights that switches between think and non-think.

DeepSeek-V3.1 introduced a hybrid reasoning architecture in a single model: DeepSeek-V3.1-Think for complex problems and a faster direct mode for the trivial, switchable by the user (InfoQ). Important for serving: reasoning is off by default and is turned on with thinking=True in chat_template_kwargs.
Qwen3 brings Hybrid Thinking Modes and an explicit Thinking Budget to bound the reasoning compute; in its series thinking is on by default (like QwQ-32B).
gpt-oss shows a different reasoning structure from the R1/Qwen family, which in practice means that the parser and the delimiters are not universal: each family has its own.

The architectural consequence is clear: in 2026 you no longer decide “a model that reasons” vs “one that does not”, but when you let the same model reason. That turns reasoning into a routing and budgeting decision, not a model-catalogue one. It connects directly with the L7 inference router: the “cheap” route turns thinking off; the “expensive” one turns it on with a cap.

The economics: why the scratch paper drives up the bill

Here is the heart of the problem. Reasoning tokens are invisible to the user but real in compute and in billing. Providers charge them as output tokens; API providers bill those reasoning tokens at the output price, not for free (Test-Time Compute Quietly Changed the Economics of Inference). On-premise there is no external invoice, but there is a cost: they occupy batch slots, KV VRAM and watts.

The billable cost (or the equivalent cost in GPU·hours) of a request is:

$$\text{compute tokens} = \text{prompt} + \underbrace{\text{reasoning tokens}}_{\text{hidden, variable}} + \text{visible answer}$$

The middle term is the one that runs out of control. On hard tasks the reasoning/visible-answer ratio can be enormous, and empirical analysis has measured up to 113× the energy cost against standard inference on certain coding tasks (Towards Data Science · Inference Scaling).

And it does not always buy accuracy. The overthinking phenomenon is documented: more thinking does not guarantee a better answer, and beyond a certain point it makes things worse. There is work showing that shorter chains are up to 34.5 % more accurate than the longest one sampled for the same question (Don’t Overthink it), plus studies on the “mirage” of test-time scaling (Mirage of Test-Time Scaling) and on when more thinking hurts (When More Thinking Hurts). The first large-scale study of test-time scaling, more than 30 billion tokens generated with 8 open-source LLMs from 7B to 235B across 4 reasoning datasets, systematises where it scales and where it does not (The Art of Scaling Test-Time Compute).

The operational reading: reasoning is a cost you have to budget for, not a “more quality” switch you leave on full all the time.

Controlling the thinking budget

Three mechanisms, from coarsest to finest.

1. Switching think / non-think

The cheapest thing is not to reason when it is not needed. On hybrid models, turn thinking off for classification, extraction, formatting, trivial answers:

# DeepSeek-V3.1: reasoning OFF by default; turned on explicitly
chat_template_kwargs = {"thinking": True} # only on the "hard" route

This is 80 % of the saving for 20 % of the effort: route by difficulty and reserve thinking for what actually pays for it.

2. Setting a cap: `thinking_token_budget`

vLLM ships a sampling parameter thinking_token_budget that counts reasoning tokens and, on reaching the limit, forces the block to close (reasoning_end_str, typically </think>), making the model answer straight away. If it is not specified, there is no limit other than max_tokens (PR #37112, implementation note). Underneath, a ReasoningBudgetLogitsProcessor injects something like “Let me stop thinking and answer now.</think>” when the cap is reached.

# Define the delimiters of the reasoning block
vllm serve Qwen/Qwen3-32B \
 --reasoning-parser qwen3 \
 --reasoning-config '{"reasoning_start_str":"<think>","reasoning_end_str":"</think>"}'

// per request: cut the scratch paper off at 1024 tokens
{"messages":[...], "thinking_token_budget": 1024}

3. Budget forcing (state of the art)

The technique that popularised fine-grained control is budget forcing, from the s1 paper: to spend less, thinking is terminated by forcing the end delimiter; to spend more, the end is suppressed and the word “Wait” is appended, which pushes the model to review and often correct its own reasoning. With this, s1-32B (a Qwen2.5-32B-Instruct fine-tuned on just 1,000 examples) beat o1-preview by as much as 27 % on AIME24/MATH (s1: Simple Test-Time Scaling). The serving lesson: the budget is not only a cost cap, it is a bidirectional quality lever.

$$\text{accuracy} \approx f(\text{reasoning tokens}) \quad\text{with a maximum, not monotonic}$$

The curve rises, saturates and on many tasks falls. Your job is to operate it near the knee, not at the extreme.

The impact almost nobody measures: the KV cache

This is where reasoning really bites serving. Every token of scratch paper is one more token in the KV cache. A long reasoning chain inflates the working memory just like a long input context.

The KV size grows linearly with the sequence (reasoning included):

$$\text{KV bytes} = 2 \times L \times h_{kv} \times d_{head} \times s \times b$$

where $L$ is layers, $h_{kv}$ KV heads (few, with GQA), $d_{head}$ the per-head dimension, $s$ the sequence length and $b$ the bytes per element. The term reasoning blows up is $s$: a “normal” 500-token answer and one with 8,000 tokens of thinking do not take up the same space, not by a long way.

The real numbers are frightening: for a distilled reasoning model with standard attention such as DeepSeek-R1-Distill-Llama-70B, a single 30,000-token reasoning chain consumes ~9 GB of KV in FP16, which quantising the KV to FP8 reduces to ~4.5 GB, one of the highest-impact optimisations for chain-of-thought workloads (Spheron · KV Cache Optimization). Multiply that by concurrency and you understand why a cluster that served 200 simultaneous requests of a “normal” model drops to 40 with the same model reasoning.

Two state-of-the-art nuances worth knowing:

Prefix caching is not free here. The empirical study of reasoning-model serving finds that prefix caching clearly improves speed on models of 14B or more, but hurts on 7B models (cache management costs more than it saves) (Reasoning Language Model Inference Serving Unveiled). And since the bulk of the reasoning cost sits in the decode (generating tokens one by one), prefix caching, which only speeds up the prefill, barely touches the expensive phase. It connects with prefix cache: engineering the hit rate.
Reasoning-specific KV compression. Techniques such as R-KV have appeared, compressing the KV by exploiting the redundancy typical of chains of thought (the model repeats itself and rambles), recovering memory without touching quality (R-KV).

The most profitable and available lever today is still KV in FP8 (see FP8 end to end) combined with a reasoning budget that avoids 30k-token tails in the first place.

Reasoning + structured output: order matters

If you serve structured output (JSON with a schema) and reasoning at the same time, there is a subtle trap: the grammar must not be applied during the thinking block. The model needs free text to reason; forcing the schema inside the <think> breaks the reasoning. vLLM solves it by having the structured-output engine (xgrammar) use the reasoner’s end_token_id to skip the constraint while the block lasts and apply it only to the final answer (vLLM · Reasoning Outputs). If you build this by hand on another engine, remember: first let it think, then impose the form. It links with structured output.

Reference architecture

On the blog’s example cluster, a node of 4×H100 SXM (80 GB, NVLink), a sensible topology for serving reasoning without wrecking the SLO:

A cheap difficulty classifier up front (a small model or rules) that picks the route. Most of the traffic does not need to think.
Fast route (non-think): the same hybrid model with reasoning off. Low TTFT, predictable cost.
Reasoning route (think): reasoning on with thinking_token_budget by default (say 1–2k tokens) and the option to raise it per request for the genuinely hard cases.
KV in FP8 in the engine on the reasoning route, plus per-request reasoning-length metrics exported to your observability (instrumenting vLLM with OTel).

To prototype this routing outside the cluster, an RTX 5090 (Blackwell, 32 GB) serves a 7–14B reasoning model and lets you validate parsers, budgets and the reasoning_content/content separation before touching the H100s. Do not expect to serve a 32B in think mode at high concurrency on a consumer card: the reasoning KV eats the 32 GB in no time.

Sizing starts, as always, from an SLO; the difference is that now the “answer size” in your capacity planning has to count the scratch paper, not just the visible answer.

Operational pitfalls (and honest scepticism)

Leaking reasoning_content. If the gateway does not separate it, you end up showing or logging the scratch paper, which may contain wrong attempts, sensitive data or language you do not want in your UI. Always separate it at L7.
Serving reasoning “just in case”. Overthinking is real and sometimes lowers accuracy. Do not turn think on by default: turn it on per route, with a budget.
Forgetting the KV. The most common mistake: sizing concurrency from the length of the visible answer. The KV is dictated by the total sequence, reasoning included. Measure p50/p95 reasoning tokens per endpoint.
Prefix caching as a silver bullet. On small models it can make things worse, and in any case it does not touch the decode, which is where the reasoning cost goes.
Assuming universal delimiters. R1, Qwen3 and gpt-oss do not share a reasoning structure. Use the right --reasoning-parser per family and test streaming (the parser has to separate correctly token by token).
Believing more compute = more intelligence. The state of the art (s1, short-m@k, “When More Thinking Hurts”) points the other way in many cases: there is a knee, and operating beyond it is burning watts. Test-time compute is a tool with diminishing, and sometimes negative, returns, not a linear lever.

A note of caution for June 2026: the field of reasoning control (budget forcing, reasoning-specific KV compression, thinking early-exit) moves fast and almost all of it is very recent. The specific figures, the 113×, the 34.5 %, the 9 GB, depend on model, task and configuration; take them as an order of magnitude for design, not as constants. Measure on your own workload before committing to SLOs.

Closing

Serving a reasoning model means serving a model whose cost per request you decide with the thinking budget, not the catalogue. The chain goes: route by difficulty → turn think off where it adds nothing → set thinking_token_budget by default on the route that thinks → quantise the KV to FP8 → measure reasoning length as a first-class metric. Do that, and reasoning becomes a capability you pay for only when it buys accuracy. Do not, and your cluster will serve a fifth of the users while the model fills sheets of scratch paper nobody is going to read.

Sources

vLLM · Reasoning Outputs — https://docs.vllm.ai/en/stable/features/reasoning_outputs/
vLLM PR #37112 · reasoning_budget — https://github.com/vllm-project/vllm/pull/37112
Adding Reasoning Budget to vLLM — https://shuyo.wordpress.com/2026/03/26/adding-reasoning-budget-to-vllm-reasoning-token-limit/
s1: Simple Test-Time Scaling — https://arxiv.org/abs/2501.19393
The Art of Scaling Test-Time Compute — https://arxiv.org/abs/2512.02008
Don’t Overthink it (short-m@k) — https://arxiv.org/pdf/2505.17813
Mirage of Test-Time Scaling — https://arxiv.org/pdf/2506.04210
When More Thinking Hurts — https://arxiv.org/html/2604.10739v1
Reasoning Language Model Inference Serving Unveiled — https://arxiv.org/pdf/2510.18672
R-KV: Redundancy-aware KV Cache Compression — https://arxiv.org/pdf/2505.24133
Spheron · KV Cache Optimization Guide (2026) — https://www.spheron.network/blog/kv-cache-optimization-guide/
Test-Time Compute Quietly Changed the Economics of Inference — https://medium.com/@Elongated_musk/test-time-compute-quietly-changed-the-economics-of-inference-9d0b8d77641c
Inference Scaling (Test-Time Compute) · Towards Data Science — https://towardsdatascience.com/inference-scaling-test-time-compute-why-reasoning-models-raise-your-compute-bill/
DeepSeek-V3.1 Hybrid Reasoning · InfoQ — https://www.infoq.com/news/2025/09/deepseek-v31-hybrid/

Test-Time-Compute on lo0 — Tech Blog