LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

· By José Manuel Román
Contents

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
  requests: { cpu: "1", memory: "4Gi" }
  limits:   { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
  disable_error_logs: true      # stops writing provider errors
  proxy_batch_write_at: 60      # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
  use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

EndpointAuthenticationWhat it doesWhat it is for
/health/livelinessNoReturns I'm alive!, or 503 during shutdownlivenessProbe
/health/readinessNoState of the process and the database. 503 if the configured Postgres does not answerreadinessProbe
/healthYesFires a real request against every model in the catalogueManual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
  background_health_checks: true
  health_check_interval: 300          # seconds
  background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

ConstantValueWhat it governs
DEFAULT_MAX_RETRIES2Retries before moving on to fallbacks
DEFAULT_ALLOWED_FAILS3Failures before cooling down a deployment
DEFAULT_COOLDOWN_TIME_SECONDS5Seconds a deployment stays out
SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD1000Minimum requests before applying the cooldown logic with a single deployment
DEFAULT_REQUEST_TIMEOUT_SECONDS6000.0Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
  num_retries: 2
  timeout: 120
  fallbacks: [{"llama-70b": ["qwen-30b"]}]
  context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
  content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
  request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

  1. --num_workers 1 and scaling by replicas.
  2. requests equal to limits, 1 vCPU and 4 GiB per worker.
  3. HPA on CPU at 60%. Never on memory.
  4. database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
  5. Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
  6. livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
  7. startupProbe with a margin of up to 300 seconds.
  8. request_timeout: 600.
  9. LITELLM_LOG out of DEBUG, and disable_error_logs: true.
  10. A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

See also

Sources