--- title: "Virtual keys, budgets and limits in LiteLLM: the layer that decides who gets the GPU, and the four things the documentation gets wrong" url: https://blog.lo0.es/en/posts/litellm-virtual-keys-budgets-limits/ markdown: https://blog.lo0.es/en/posts/litellm-virtual-keys-budgets-limits/index.md lang: en date: 2026-09-08 lastmod: 2026-09-13 author: "José Manuel Román" description: "LiteLLM Proxy access control as of September 2026, read from the code and not from the documentation: the organisation, team, user and end-customer hierarchy and the rule that breaks it, the life cycle of a virtual key with its rotation and its grace period, the v3 limiter that is already the default, the 429 the documentation still calls a 400, the audit table with its six actions and its different default in OSS and in enterprise, and what can be demonstrated to an ENS or ISO/IEC 42001 auditor with all of this in place." tags: ["litellm","gateway","multi-tenancy","governance","budgets","rate-limiting","keycloak","oidc","audit","ens","iso-42001","llmops","on-premise"] translations: es: https://blog.lo0.es/posts/litellm-limites-presupuestos-claves-virtuales/index.md --- # Virtual keys, budgets and limits in LiteLLM: the layer that decides who gets the GPU, and the four things the documentation gets wrong > Third article in the operational track of the control layer. The [pair with Langfuse](https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/index.md) covered observability, and [day 2 of the proxy](https://blog.lo0.es/en/posts/litellm-day-2-high-availability/index.md) covered availability. Here it is governance: who can call, which model, how much and on what budget. The economic model behind those figures is in [FinOps and multi-tenancy with LiteLLM](https://blog.lo0.es/en/posts/finops-multi-tenancy-gpu-cluster-who-pays/index.md). ## TL;DR On a platform running on your own GPUs there is no provider invoice to put consumption in order. Capacity is fixed, known and shared, so someone has to impose the allocation, and that someone is the gateway. **The hierarchy has six levels**: organisation, team, team member, internal user, key and end customer, plus the project and tag scopes. Each level's budgets are evaluated **independently and all of them have to pass**. They neither add up nor cancel each other out, with one exception. **The exception is in the code and it is the most important governance trap in the system.** A key that belongs to a team suppresses the check of its owner's personal budget, unless `apply_user_budget_to_team_keys` is enabled. A user with a hundred-euro cap who works with their team's key spends whatever the team allows. **Rotating a key preserves its spend.** `/key/regenerate` does an UPDATE on the same row changing the hash, so `spend`, `max_budget` and the budget window remain intact. It accepts a grace period, stored in a table of its own, and if the period's format is wrong it is ignored without blocking anything. **The v3 limiter is already the default** in 1.100.0, and you go back to the previous one with an environment variable. Its window is sliding, sixty seconds, anchored to the first request, and the counters are incremented with Lua scripts inside Redis using Redis's own timestamp to close the race between replicas. And four points where the official documentation does not match the code, verified against tag v1.100.0: **an exhausted budget returns 429**, not 400; **the audited actions are six**, not three; `/key/info` and `/key/list` are **GET**, not POST; and the v3 limiter carries a docstring saying it is not production-ready while it is the one loaded by default. ## You are here: the gateway as an administrative boundary Without this layer, an in-house inference platform has a single mode of operation: whoever knows the URL consumes. It works while there is one team. It stops working the day there are three, or the day someone connects an agent that fires twenty calls per interaction and wrecks everyone else's time to first token. The two questions this layer answers are different and have to be kept apart. The **capacity** one is instantaneous: how many requests and how many tokens per minute each consumer can push, so the fleet does not saturate and latency stays within the service agreement. The **cost** one is cumulative: how much each consumer has spent this month against its budget. They are configured in the same place, applied at different moments and fail in different ways. ## The analogy: the building's key ring An office building hands out keys at three levels. The master opens everything and the building manager has it. The floor keys open a whole floor and each tenant company has one. And the office keys open one office. Three properties of that system are the ones that matter here. A key is replaced without changing the lock of the whole building, and the tenant keeps their parking space and their electricity meter: that is rotation preserving the spend. A lost key is cancelled from the central panel without collecting anything physical: that is blocking from the interface. And the record of who took out each key and when lives in a separate book, which is what you show an inspector: that is the audit table. The analogy breaks at one point, and that point is where this system's governance flaw lives. In the building, having a floor key does not cancel the office's consumption limit. In LiteLLM, it does. ## The hierarchy, and the line that breaks it The levels and their management endpoints: | Level | Endpoints | Budget | |---|---|---| | Organisation | `/organization/new`, `/organization/member_add`, … | `max_budget` | | Team | `/team/new`, `/team/member_add`, `/team/block`, … | `max_budget` | | Team member | `/team/member_add` with `max_budget_in_team` | Its own row in the budgets table | | Internal user | `/user/new`, `/user/update`, … | `max_budget` | | Key | `/key/generate`, `/key/update`, … | `max_budget`, `soft_budget` | | End customer | `/customer/new`, and the parallel `/end_user/*` set | `max_budget` | The default behaviour is the right one for a multi-tenant platform: each scope carries its own counter and the check of each one is independent of the rest. A request passes if all of them pass. And then there is this condition, which lives in the authorisation checks: ```python is_team_key = team_object is not None and team_object.team_id is not None if is_team_key and general_settings.get("apply_user_budget_to_team_keys") is not True: return ``` Translated: when the key belongs to a team, the check of the budget of the user who owns that key **returns without evaluating anything**. The personal cap exists in the database, is visible in the interface, and is not applied. For a deployment where personal budgets are decorative, it makes no difference. For one where the per-person cap is the control that was shown to someone as a guarantee that a user cannot overrun the platform, it is a hole. The line that closes it is one: ```yaml general_settings: apply_user_budget_to_team_keys: true ``` ## Virtual keys: the complete life cycle The endpoints all exist and some of them are not where you expect: | Route | Method | What for | |---|---|---| | `/key/generate` | POST | Create | | `/key/update` | POST | Modify | | `/key/info` | **GET** | Query one | | `/key/list` | **GET** | List | | `/key/delete` | POST | Delete | | `/key/regenerate` | POST | Rotate | | `/key/block`, `/key/unblock` | POST | Suspend and reactivate | | `/key/health` | POST | Check that key's callbacks | The two marked in bold are GET, and several examples in the documentation suggest otherwise. `/key/health` is not what its name suggests either: it does not validate the key or its budget, it checks the logging callbacks associated with it. There are many creation parameters, and these are the ones that govern: ```bash curl -X POST 'https://gateway.interno/key/generate' \ -H 'Authorization: Bearer sk-...' \ -d '{ "key_alias": "equipo-datos-notebooks", "team_id": "t-datos", "models": ["llama-70b", "qwen-30b"], "max_budget": 250, "budget_duration": "30d", "tpm_limit": 400000, "rpm_limit": 600, "max_parallel_requests": 8, "duration": "90d", "tags": ["produccion", "notebooks"] }' ``` Alongside those there are controls that are rarely used and solve specific problems: `model_max_budget` to set a per-model cap within the same key, `model_tpm_limit` and `model_rpm_limit` for the same thing with capacity, `enforced_params` to require certain keys to always arrive with given fields, `allowed_routes` so an application key cannot touch the management endpoints, and `blocked` to be born suspended. ### How they are stored A key's hash is a single-pass hexadecimal SHA-256, with no salt and no derivation function: ```python hashed_token = hashlib.sha256(token.encode()).hexdigest() ``` The row lives in `LiteLLM_VerificationToken`, with the hash as primary key, and it contains the accumulated spend, the limits, the expiry, the allowed models and the rest of the configuration. Two consequences to be clear about before an audit. The first is that this hash is not designed to resist a dictionary attack against the database, and the code itself acknowledges it in a comment about encryption key derivation, where it admits that a single-pass unsalted SHA-256 is not a derivation function and that moving to HKDF would be more defensible in an audit. The real mitigation is that virtual keys are generated with enough entropy, not the strength of the hash. The second is that `LITELLM_SALT_KEY` **plays no part here**: that variable encrypts the stored provider credentials, not the virtual keys. There is a trap around that variable that takes down a whole deployment. If it is not defined, it **falls back to the master key**. Rotating `LITELLM_MASTER_KEY` without having first set your own `LITELLM_SALT_KEY` leaves every stored provider credential unreadable, and the failure does not block: it logs an error and returns null, so the symptom shows up as models that stop authenticating with no explanation. Setting `LITELLM_SALT_KEY` on day one, and never touching it, is one of the things that saves the most grief. ### Rotation `/key/regenerate` does not create a new key and delete the old one. It updates the same row changing the hash, so **the accumulated spend, the maximum budget and the reset window are preserved**. For a monthly chargeback this is exactly what you want: rotating mid-month does not zero the counter. It also accepts a grace period, so the old key remains valid while consumers update: ```bash curl -X POST 'https://gateway.interno/key/sk-vieja/regenerate' \ -H 'Authorization: Bearer sk-...' \ -d '{"grace_period": "24h"}' ``` The old hash is stored in a separate table with its revocation date. Without that parameter, revocation is immediate, and the default value can be set with `LITELLM_KEY_ROTATION_GRACE_PERIOD`. A detail with teeth: if the period's format is wrong, the system logs a warning and **continues with no grace period**. The rotation completes, the old key dies on the spot, and all that is left is a log line. In an automated rotation procedure, that combination of "carry on" and "warn in the log" is what produces the outage at three in the morning. ## What exactly happens when a budget runs out The code raises `BudgetExceededError` with **status 429**. The documentation says 400 in one place, and there the code wins. Its being a 429 has a practical implication: OpenAI clients retry automatically on a 429, so an exhausted budget turns into a retry storm against a gateway that is going to keep rejecting. Internal clients should distinguish the reason, which travels in the message. The messages are templates and state the scope, the spend and the cap: ``` Budget has been exceeded! Key= Current cost: , Max budget: Budget has been exceeded! Team= Current cost: , Max budget: ExceededBudget: User= over budget. Spend=, Budget= ExceededBudget: End User= over budget. Spend=, Budget= Budget has been exceeded! Organization= ... Budget has been exceeded! Tag= ... ``` Two clarifications about the comparison. It is `spend >= cap`, with the equals sign included, so landing exactly on the cap already blocks. And the soft budget, `soft_budget`, **blocks nothing**: it only writes a log line when it is crossed. It serves to warn, not to contain, and building an alert on top of it is the right way to use it. The rest of the rejections, with their codes: | Situation | Code | Behaviour | |---|---|---| | Budget exhausted | 429 | Message with scope, spend and cap | | Expired key | 401 | It also deletes the cache entry, so the expiry takes effect across the whole fleet without waiting for the TTL | | Model not allowed | 403 | Different error type depending on the scope: key, team, user, organisation or project | | Blocked key | Generic exception | `Key is blocked. Update via /key/unblock if you're an admin.` | The last row is an inconsistency in the project worth knowing about: blocking raises an exception with no explicit type or code, while expiry and model access do carry them. Any alert that discriminates by error type will not see blocks the way it sees the rest. A useful note about allowed models: the check accepts both the alias and the underlying model, so a key authorised for `llama-70b` can call it by its alias or by its real name with no additional configuration. ## Capacity limits: the v3 limiter This is the most significant silent change of the year in this layer. In 1.100.0, the limiter loaded by default is the third version. Going back to the previous one has to be asked for explicitly: ```bash LEGACY_MULTI_INSTANCE_RATE_LIMITING=true ``` There is no flag to turn it on, only to turn it off. And it carries a contradiction worth knowing before trusting it: the file itself still opens with a comment saying it is under development and not production-ready, while it is the one running. The design is avowedly inspired by Envoy's. **The window is sliding and lasts sixty seconds**, configurable with `LITELLM_RATE_LIMIT_WINDOW_SIZE`. It is not an exact rolling log nor a clock-aligned window: it anchors to the first request and resets once the window size has elapsed from that moment. For one-minute limits, the difference from a calendar window shows up at the edges and rarely matters. What does matter is Redis. The counters are incremented with Lua scripts executed inside Redis, using Redis's own timestamp and not the pod's, precisely so window resets are deterministic across replicas and to close the race between checking and counting. Without Redis, **each pod counts on its own and the effective limit is multiplied by the number of pods**. A limit of 600 requests per minute with five replicas is a limit of 3,000. There is an inherent imprecision in the token limits that Redis does not solve. The number of output tokens is not known until the response finishes, so the limiter counts with an estimate before the call and reconciles afterwards. The possible excess is, per in-flight request, the difference between the estimate and the real figure. With high concurrency and long responses, that allows the cap to be exceeded within a window. To protect the fleet, the concurrent request limit (`max_parallel_requests`) is a more direct instrument than tokens per minute. ### What the client sees When limited, a 429 with the message giving the scope, the current limit, what is left and when it resets. And three headers: `retry-after` with the window size, that is `60` by default, plus `rate_limit_type` and `reset_at`. On the success path, the informational headers use a nested format of their own: ``` x-ratelimit-api_key-remaining-requests x-ratelimit-api_key-limit-tokens x-ratelimit-team-remaining-requests ``` **They are not the standard names** `x-ratelimit-limit-requests` that OpenAI and compatibles emit, so a client expecting the usual format will find nothing. There is also an open issue about these headers being lost on streaming responses, which is most of an assistant's traffic. ## Identity: beyond the static key Handing out keys by hand works with three teams and stops working with thirty. The two routes for connecting the gateway to corporate identity: **JWT.** It is enabled with `enable_jwt_auth` and configured with the `litellm_jwtauth` block, which maps token fields to LiteLLM entities: the team comes from `team_id_jwt_field`, the user from `user_id_jwt_field`, the organisation from `org_id_jwt_field`. It supports dot notation for nested claims, automatic creation of users and teams with `user_id_upsert` and `team_id_upsert`, role-based access control with `enforce_rbac`, per-team model restriction with `enforce_team_based_model_access`, and several issuers at once. The issuer's public key is cached with a TTL of 600 seconds by default. ```yaml general_settings: enable_jwt_auth: true litellm_jwtauth: team_id_jwt_field: "groups" user_id_jwt_field: "sub" user_email_jwt_field: "email" enforce_rbac: true enforce_team_based_model_access: true team_id_upsert: true ``` **OIDC for the admin interface.** With Keycloak you use the generic provider, and the variables are `PROXY_BASE_URL`, `GENERIC_CLIENT_ID`, `GENERIC_CLIENT_SECRET`, `GENERIC_AUTHORIZATION_ENDPOINT`, `GENERIC_TOKEN_ENDPOINT`, `GENERIC_USERINFO_ENDPOINT`, plus a set of attributes to map the identifier, the email, the name and the role, and `GENERIC_ROLE_MAPPINGS_*` to translate Keycloak groups into LiteLLM roles. Login is rate-limited and sessions have a TTL of their own. For automatic provisioning there is SCIM in the enterprise version, with base `/scim/v2`, and deprovisioning a user blocks their keys and takes them out of the authentication cache. That part is not in the open tree, so it cannot be verified against the code. This fits with the identity work already covered in [hardening and secrets of the sovereign stack](https://blog.lo0.es/en/posts/hardening-secrets-sovereign-llm-stack-defence-in-depth/index.md) and in the article on [MCP authentication with Keycloak](https://blog.lo0.es/en/posts/mcp-grows-up-authentication-keycloak/index.md), which uses the same issuer. ## Audit: what gets recorded The table is `LiteLLM_AuditLog` and its columns are what you would expect from a change log: ``` id, updated_at, changed_by, changed_by_api_key, action, table_name, object_id, before_value, updated_values ``` It stores the previous value and the new one, and attributes the change to an actor and to the key it was made with. Three clarifications the documentation does not capture well. **The actions are six, in the past tense**: `created`, `updated`, `deleted`, `blocked`, `unblocked` and `rotated`. The documentation mentions three. That blocking, unblocking and rotation are actions in their own right is exactly what is needed to reconstruct a credential's life cycle for an auditor. **The scope is wider than advertised.** Besides teams, users, keys and models, changes to the proxy configuration and to the SSO configuration are audited. A change of identity issuer is recorded. **The default depends on the licence.** The resolution order is `litellm_settings.store_audit_logs`, then the variable `LITELLM_STORE_AUDIT_LOGS`, and if neither is set, it stays **enabled in enterprise and disabled in the open version**. An OSS deployment that assumes there is an audit trail does not have one: ```yaml litellm_settings: store_audit_logs: true ``` Two more pieces for a serious chain of evidence. The actor attribution can be passed with the `LiteLLM-Changed-By` header, and its use is restricted by the key's or the team's configuration, so any caller cannot forge it. And the events can be shipped out with `audit_log_callbacks` and its S3 parameters, which is what allows depositing them in an immutable store with object lock. That last part is what turns the record into evidence. The Postgres table is mutable by anyone with database access, and as of September 2026 it has no indexes, so querying it by date range over a long history will not be comfortable either. ## What this demonstrates to an auditor, and what it does not This is the part that decides whether the setup is good enough for a regulated customer. With everything above configured, the platform sustains without difficulty: **Access control via corporate identity**, with onboarding and offboarding in the identity provider and not in the gateway, and with the role determining which models are reachable. **Segregation between tenants**, with independent counters per organisation, team and person. **Traceability of administrative changes**, with actor, time, previous value and new value, exportable to an immutable store. And **demonstrable consumption limits**, with the evidence of the rejection in the spend log. What it does not cover, and has to be solved elsewhere: **The record of the interactions** is not this. The audit table records configuration changes, not model calls. The calls are in the spend log and in the traces, and traces are best-effort, which is the discussion in the [first article in the series](https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/index.md). **The integrity of the record** is not something Postgres provides. Without export to immutable storage, the evidence is only as strong as the database permissions. **The content of the prompts** is a separate problem, with its own personal-data treatment and its own decision about whether it is stored, masked or not logged at all. And a warning about the control that tends to be shown first: while `apply_user_budget_to_team_keys` is not enabled, the per-person cap is not being applied to team keys. Showing that field in the interface as proof of a control that is not executed is the kind of finding a competent auditor discovers. The complete control mapping is in [ENS, ISO 42001 and the EU AI Act](https://blog.lo0.es/en/posts/technical-controls-ens-iso-42001-eu-ai-act-cross-mapping/index.md). ## Checklist 1. `apply_user_budget_to_team_keys: true` if personal budgets have to be enforced. 2. `LITELLM_SALT_KEY` set on day one, different from the master key and never rotated lightly. 3. `store_audit_logs: true` explicitly, without trusting the default. 4. `audit_log_callbacks` towards an immutable store if an audit has to be sustained. 5. Redis mandatory if there is more than one replica, or the limits get multiplied by the number of pods. 6. `max_parallel_requests` per key, on top of the per-minute limits, to protect the fleet from a runaway agent. 7. `allowed_routes` on application keys, so they cannot reach the management endpoints. 8. `soft_budget` with an alert on top, understanding that it does not block. 9. Rotation with a verified `grace_period`, because an invalid format revokes on the spot. 10. JWT against Keycloak as soon as there are more than a handful of teams, with `enforce_rbac` and `enforce_team_based_model_access`. ## Traps and things that are not what they look like **A user's cap is not enforced.** They are using a team key and `apply_user_budget_to_team_keys` is missing. **Clients retry in a loop against an exhausted budget.** The code is 429 and the OpenAI libraries retry 429s automatically. **The soft budget has stopped nothing.** It does not block, it only logs. **The per-minute limits are enforced at double or triple.** There is no Redis, and each replica keeps its own count. **The client does not see the limit headers.** The names are nested and proprietary, and in streaming there is an open issue where they get lost. **There is no audit log.** It is an OSS deployment and the default there is disabled. **The models stop authenticating after rotating the master key.** There was no fixed `LITELLM_SALT_KEY`, and the encrypted provider credentials can no longer be decrypted. **The rotation cut the service on the spot.** The `grace_period` carried a format the system did not understand and it carried on without it. **Blocking a key does not fire the alert.** It is raised as a generic exception, without the type that expiry and model access do carry. ## Closing The keys and budgets layer is the one that turns a shared URL into a platform with tenants. It is well solved in LiteLLM and has more depth than the documentation suggests, with per-model budgets inside a key, multiple budget windows, per-tag limits and rotation that preserves the history. Out of the whole article, two things take the attention. The `apply_user_budget_to_team_keys` line, because it is the difference between having a control and believing you have one. And setting `LITELLM_SALT_KEY`, because its default turns a routine rotation of the master key into an outage that is hard to diagnose. Both are one line of configuration and both are discovered late. ## See also - [Humans and agents on the same gateway](https://blog.lo0.es/en/posts/humans-agents-same-gateway-priority-traffic-separation/index.md) — what happens to these limits when the caller is an agentic loop, and why `max_parallel_requests` is only applied on the key. - [LiteLLM and Langfuse: the operational pair](https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/index.md) — where the traces of all this end up, and why the audit record cannot lean on them. - [LiteLLM on day 2: high availability](https://blog.lo0.es/en/posts/litellm-day-2-high-availability/index.md) — the Redis taken as given here, and the Postgres arithmetic that sustains the spend table. - [FinOps and GPU multi-tenancy with LiteLLM](https://blog.lo0.es/en/posts/finops-multi-tenancy-gpu-cluster-who-pays/index.md) — where the figures that go into `max_budget` come from, and the difference between showback and chargeback. - [From GPU-hour to cost per token and per request](https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/index.md) — the conversion that gives meaning to a budget in euros on your own GPU. - [Technical controls: ENS, ISO 42001 and the EU AI Act](https://blog.lo0.es/en/posts/technical-controls-ens-iso-42001-eu-ai-act-cross-mapping/index.md) — the complete mapping, of which this article covers the access control and administrative traceability part. - [Hardening and secrets of the sovereign stack](https://blog.lo0.es/en/posts/hardening-secrets-sovereign-llm-stack-defence-in-depth/index.md) — the management of `LITELLM_SALT_KEY` and of the rest of the deployment's secrets. - [When MCP grows: authentication with Keycloak](https://blog.lo0.es/en/posts/mcp-grows-up-authentication-keycloak/index.md) — the same identity issuer, applied to the platform's other front door. - [Keycloak in an AI platform](https://blog.lo0.es/en/posts/keycloak-identity-plane-ai-platform/index.md) — the alternative to these keys, and why a long-lived key decouples identity from the provider. ## Sources - LiteLLM, *Virtual Keys* (parámetros de creación, ciclo de vida, rotación): . - LiteLLM, *Budgets, Rate Limits* (jerarquía, presupuestos por nivel, clientes finales): . - LiteLLM, *JWT Token Auth* (bloque `litellm_jwtauth` y todos sus campos): . - LiteLLM, *Audit Logs* y administración múltiple (la cabecera `LiteLLM-Changed-By`): . - LiteLLM, *SCIM* (aprovisionamiento automático, versión enterprise): . - LiteLLM, `litellm/proxy/auth/auth_checks.py` en la etiqueta v1.100.0 (la condición de `apply_user_budget_to_team_keys` y los mensajes de presupuesto agotado): . - LiteLLM, `litellm/proxy/management_endpoints/key_management_endpoints.py` en v1.100.0 (rutas, rotación sobre la misma fila, periodo de gracia): . - LiteLLM, `litellm/proxy/hooks/parallel_request_limiter_v3.py` y `litellm/proxy/hooks/__init__.py` en v1.100.0 (el limitador v3 como valor por defecto, la ventana, los scripts Lua): . - LiteLLM, `litellm/exceptions.py` en v1.100.0 (`BudgetExceededError` con código 429): . - LiteLLM, `litellm/proxy/common_utils/encrypt_decrypt_utils.py` en v1.100.0 (`LITELLM_SALT_KEY` cayendo en la clave maestra, y el comentario sobre la derivación de claves): . - LiteLLM, `schema.prisma` en v1.100.0 (`LiteLLM_VerificationToken`, `LiteLLM_AuditLog`, la tabla de claves deprecadas): . - LiteLLM, issue #27748 — cabeceras `x-ratelimit-*` perdidas en respuestas en streaming con el limitador v3: .