Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

1. Model calls per hour of human work, by user profile.
2. Prompt length distribution (median and 95th percentile), not the mean.
3. Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

• 98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
• 55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
• 8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

1. Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
2. Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

• Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
• Subagents: 44 % of sessions, median of 4 deployments (AgentX).
• Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

• The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
• The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
• The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
• The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
• A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

• vllm:num_requests_running, vllm:num_requests_waiting
• vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
• vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

• vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
• The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
• vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
• vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

1. 90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
2. 90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference config.yaml

model_list:
  - model_name: agentes-30b
    litellm_params:
      model: hosted_vllm/qwen3-coder-30b
      api_base: http://vllm-agentes.inferencia.svc:8000/v1
      rpm: 600
    model_info:
      input_cost_per_token: 0.00000018
      output_cost_per_token: 0.00000072

router_settings:
  routing_strategy: simple-shuffle
  # without this, the capacity calculation in this article is worthless
  optional_pre_call_checks: ["session_affinity"]
  num_retries: 2
  allowed_fails: 3
  cooldown_time: 30

general_settings:
  # admission control: below this the queue grows without limit
  max_in_flight_requests_per_worker: 24
  max_queued_requests_per_worker: 48
  admission_queue_timeout_seconds: 1.0
  # the spend queue discards at 64 MB; shorter interval, less backlog
  proxy_batch_write_at: 60
  enable_drain_endpoint: true
  drain_endpoint_token: os.environ/DRAIN_TOKEN
  database_connection_pool_limit: 10

litellm_settings:
  # 6000 s by default: a hung connection holds a slot for an hour and a half
  request_timeout: 600
  callbacks: ["langfuse_otel", "prometheus"]
  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8   # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --long-prefill-token-threshold 512 \
  --max-num-queued-reqs 64 \
  --gpu-memory-utilization 0.90

Checklist

1. Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
2. Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
3. Set --max-num-seqs from that budget, not from a chat heuristic.
4. Switch on session affinity before accepting the calculated number of GPUs.
5. Switch on admission control in both layers and check that the client respects retry-after.
6. Lower request_timeout from 6,000 to 600.
7. Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
8. Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
9. Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
10. Go over the dashboards for the vLLM metric renames before upgrading.
11. Check that no routing policy moves a live session between model groups.
12. Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

See also

• Humans and agents on the same gateway: coexistence on a fleet that is already sized.
• Prefix routing: what LiteLLM does not do: the affinity mechanism on which this article’s entire calculation depends.
• LiteLLM’s MCP gateway: the other face of the tool catalogue, the one paid in tokens.
• Capacity planning for on-premise LLM inference: the classic method whose inputs this article replaces.
• KV cache: fundamentals and prefix cache hit rate engineering: the two pieces that dominate the budget.
• Day 2 of the proxy: availability, draining and deployments.

Sources

• Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
• Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
• From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
• vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
• vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
• LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
• Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
• Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.