Router on lo0 — Tech Blog

Choosing the switchboard: which OSS gateway to put in front, and why the licence decides before the features

Sat, 06 Jun 2026 08:30:00 +0200

This post is the decision companion to The LLM inference router. That one explained what an inference router is and why it exists (catalogue, traffic splitting, cross-cutting policy, prefix-aware routing). This one answers the question that actually blocks a deployment: which one to choose, with verified licences and an order of criteria that is not the one most people use.

TL;DR

The gateway is the piece 100 % of the traffic passes through and the most expensive to replace later: the client SDK, the auth and rate-limit policies, the tracing and the model catalogue all couple to it. That is why the order of the criteria matters, and it is not the usual one. First the licence: not “is it open source?” but “are the features I need under a permissive licence or behind an Enterprise wall?”. Second the architectural fit: is it a citizen of your Kubernetes (Gateway API) or a separate process you have to operate on the side? Third, maturity and documentation. Fourth, the features. With data verified as of June 2026: LiteLLM is MIT in the core, but SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise; Kong has an Apache 2.0 core but the AI plugins that matter (semantic cache, prompt guard, AI proxy advanced) are gated; Envoy AI Gateway, Gateway API Inference Extension (GIE), Higress, APISIX and Bifrost are Apache 2.0 end to end. For an RKE2 + vLLM stack with a K8s-native priority (the case we assume here), the recommendation is to adopt the Gateway API Inference Extension model, the Endpoint Picker that routes with awareness of prefix cache, KV and LoRA, which is exactly what multiplies the hit rate. Implement it with Envoy AI Gateway if the AI-native trajectory weighs more, or with Higress if today’s maturity weighs more; and put LiteLLM (MIT) behind it as an optional multi-provider control plane. With explicit scepticism about v0.x releases, about “OpenAI-compatible ≠ inference-aware” and about phone-home telemetry.

The principle that reorders the criteria: the gateway is a marriage, not a date

Some pieces of the stack can be swapped on a Sunday afternoon: the embedding model, the reranker, even the inference engine behind an OpenAI-compatible interface. The gateway cannot. It is the piece everything else couples to: every client SDK points at its URL, the security policies live in it, the tracing is born in it, it defines the model catalogue. Ripping it out two years later means touching every consumer at once. It is the infrastructure decision with the highest cost of reversal in the whole serving layer.

When a decision is expensive to reverse, the dominant criterion is not “what does it do today?” but “can I own it and operate it for years without surprises?”. And that puts the licence ahead of the features. A brilliant tool whose SSO, audit log or RBAC live behind an Enterprise contract is a tool that, the day your ENS deployment needs those controls, forces you to pay or to migrate, which is exactly the scenario the choice was supposed to avoid. That is why this post’s filter order is licence → fit → maturity/docs → features, and not the other way round.

The field of candidates (June 2026)

The real OSS pieces anyone would put in front of an on-premise vLLM fleet:

Candidate	What it is	Language
LiteLLM Proxy	OpenAI-compatible gateway, 100+ providers, virtual keys, spend	Python (FastAPI)
Envoy AI Gateway	AI layer on top of Envoy Gateway; integrates GIE/EPP, InferencePool	Go (Envoy)
Gateway API Inference Extension (GIE)	K8s-SIG extension: inference-aware Endpoint Picker	Go
Higress	API gateway + AI plugins, Envoy/Istio, CNCF	Go/C++
Apache APISIX	Mature API gateway with AI plugins	Lua/Nginx
Kong (+ AI plugins)	API gateway; Apache 2.0 core, Enterprise AI plugins	Lua/Nginx
Bifrost	High-performance AI-first gateway	Go
vLLM Production Stack / Semantic Router	vLLM-specific router, KV/prefix/intent-aware	Go/Python

Filter 1 — Licence: where the small print is

This is where half the options die, and where “open source” misleads if you do not look at the detail. What has been verified:

Candidate	Core licence	What is gated (paid)
LiteLLM	MIT	SSO (Okta/Azure AD), audit logs, JWT auth, fine-grained RBAC, several guardrails (llmguard, llamaguard, prompt-injection) → Enterprise (~250 USD/month and rising)
Envoy AI Gateway	Apache 2.0	— (full AI layer is OSS)
GIE (Inference Extension)	Apache 2.0	— (K8s-SIG project)
Higress	Apache 2.0	— (AI plugins included)
APISIX	Apache 2.0	— (more built-in AI than Kong OSS)
Kong	Apache 2.0 (core)	AI Semantic Cache, AI Prompt Guard, AI Proxy Advanced, AI RAG Injector → Enterprise
Bifrost	Apache 2.0	—

Two conclusions that settle the field:

Kong falls out for an OSS AI deployment. Its core is Apache 2.0, but precisely the AI plugins that would justify choosing it (semantic cache, prompt guard, proxy advanced) are Enterprise, with contracts that the sector’s own comparisons put above five figures a year. For AI on an OSS budget, APISIX offers more built-in at no cost. Kong remains excellent as a classic API gateway; as an OSS AI gateway, no.
LiteLLM really is MIT in the core, and that is genuine: you can fork it, modify it and use it commercially. But SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise. Under the criterion we set (“a permissive core is enough, governance is solved with OIDC and external auditing in the stack”) LiteLLM stays in play; under an “all OSS or nothing” criterion it would be wounded. It pays to know exactly what falls on which side of the line before committing.

LiteLLM: what is gated and its OSS equivalent

The nuance that decides whether LiteLLM’s gating is a real problem or not: “paid built-in feature” is not the same as “capability impossible in OSS”. In almost every case the capability is achieved by changing the how, wiring OSS through LiteLLM’s open hook, or solving it in the layer next door (the gateway, the observability). The breakdown, verified against the docs:

Capability	In LiteLLM Enterprise	OSS equivalent
Guardrails	prebuilt integrations (`llmguard`, `llamaguard`, `lakera`, `aporia`, `hide_secrets`)	the framework + custom guardrail hook + Presidio are OSS (MIT core); you invoke LLM Guard, Llama Guard or Presidio from the hook yourself
Audit log	turnkey UI with retention policies	request/response logging + custom callbacks + OTel/Langfuse exporters (official OSS integration) → you build the trail and you own it
RBAC	fine-grained (`enforce_rbac`, org/team/user roles)	coarse (virtual keys per team/budget/model) is OSS; fine-grained is done at the edge in the gateway (`ext_authz` + OPA)
SSO	Admin UI SSO (Okta/Azure/Google/OIDC)	user/API SSO is solved by fronting with OSS OIDC (Keycloak + oauth2-proxy) or in the gateway itself; JWT auth + JWT→virtual-key mapping are already in the core

The conclusion that closes the decision: in a K8s-native design with a gateway in front, OIDC and authorisation live in the gateway (Envoy/Higress + OPA) and the audit trail in Langfuse/OTel, both Apache 2.0 and already in the stack, so LiteLLM placed behind barely needs its Enterprise tier: the governance pieces sit where they architecturally belong and happen to be OSS. The only real cost is the DIY integration and maintenance, and, for ENS, designing the guarantees of the audit trail (retention, immutability/WORM) is on you, it does not come out of the box.

The rest, Envoy AI Gateway, GIE, Higress, APISIX and Bifrost, pass the licence filter clean: Apache 2.0 end to end, with no critical feature behind a wall.

Filter 2 — Fit: Kubernetes citizen or tenant?

With a K8s-native priority (RKE2), the question is whether the gateway is modelled as cluster resources, versioned with GitOps, observed with the same tools and integrated with the scheduler, or whether it is a separate process you have to operate on the side. This is where the structural novelty of 2025-2026 appears: the Gateway API Inference Extension (GIE).

GIE is an extension of the standard Kubernetes Gateway API, from SIG-Network, which adds two pieces: the InferencePool (a pool of replicas of a model as a native resource) and the Endpoint Picker (EPP), a scheduler that decides which replica each request goes to according to the inference state: queue length, available LoRA adapters, and, the key piece, the prefix cache state of each replica. It is exactly the prefix-aware routing that the router post explained as the reason hit rate goes from 5-15 % to 60-85 %, now as a community standard instead of a proprietary feature of each product.

Why this settles the decision when the priority is K8s-native: GIE turns inference-aware routing into a standard interface that several implementations satisfy (Envoy AI Gateway, Higress, kgateway, Istio, GKE Inference Gateway). You pick an implementation today and, if you change it tomorrow, the resource model (InferencePool, HTTPRoute) is preserved. That is precisely the property you wanted from a decision that is expensive to reverse: the coupling is to a standard, not to a product. A standalone proxy like LiteLLM, however good it is, lives outside this model. It is one more Deployment, with its own config, its own catalogue format and its own management plane.

Filter 3 — Maturity and documentation: the uncomfortable honesty

Here comes the tension no marketing comparison admits: what is K8s-native-correct and what is battle-tested-today do not quite coincide in June 2026.

Envoy AI Gateway: the AI layer is at v0.5, pre-1.0 and young. But it runs on Envoy and Envoy Gateway, which are among the most mature things that exist in proxies. It has the most complete GIE/EPP integration, model virtualisation and OpenInference tracing. Risk: the AI layer still moves fast (breaking changes are possible). Docs are growing, and good.
GIE: the project is reaching GA (one of the first standardised inference-aware extensions). Standard maturity is high and rising; the maturity of each implementation varies.
Higress: the most mature today among the K8s-native Apache-2.0 options with AI. Production at Alibaba scale, CNCF, Envoy/Istio base, AI plugins included, Gateway API support. If “mature” weighs more than “cutting-edge AI-native”, it is the safe bet.
APISIX: a very mature classic gateway, with growing AI plugins; less specialised in inference-aware routing (prefix/KV) than the GIE model. Battle-tested at massive scale.
LiteLLM: the one with the best documentation and the widest adoption in the field, by a distance. Mature as a proxy. Its ceiling is the language (Python/FastAPI: ~250-300 RPS per instance, scaling out by replicas) and the governance gating already covered.
Bifrost: Apache 2.0, Go, the highest performer (~11 µs overhead at 5,000 RPS), with semantic caching and governance built in. Younger and less proven over years, but technically strong.

Decision matrix

Weighting the four filters for the case of RKE2 + vLLM, K8s-native priority, permissive core acceptable:

Candidate	Licence	K8s-native	Maturity	Docs	Inference-aware	Verdict
Envoy AI GW + GIE/EPP	✅ Apache 2.0	✅✅ standard	⚠️ v0.5 (mature Envoy)	✅ good	✅✅ prefix/KV/LoRA	Primary (trajectory)
Higress (+ GIE)	✅ Apache 2.0	✅✅	✅✅ production	✅ good	✅ via GIE/plugins	Primary (maturity today)
APISIX	✅ Apache 2.0	✅	✅✅	✅✅	⚠️ less specialised	Solid alternative
LiteLLM	⚠️ MIT core, gov. gated	❌ separate proxy	✅✅	✅✅✅	⚠️ basic	Control plane behind
Bifrost	✅ Apache 2.0	⚠️	⚠️ young	✅	✅	One to watch (perf)
Kong	⚠️ AI plugins Enterprise	✅	✅✅	✅✅	❌ gated	Ruled out (OSS AI)

The recommendation

For the case we fixed, RKE2 + vLLM, K8s-native priority, permissive licence with a sufficient core, the recommendation has two layers, not one:

Data layer (what sits in front of the replicas): adopt the Gateway API Inference Extension model. It is the decision that ages well because it couples you to an Apache 2.0 standard, not to a product, and because it brings the prefix/KV/LoRA-aware routing that really moves the hit rate. For the implementation:

If the AI-native trajectory weighs more and you accept operating a v0.x layer on top of a mature Envoy: Envoy AI Gateway + GIE/EPP. That is where the cutting edge and the most complete integration are.
If today’s maturity outweighs everything else (ENS, critical production, zero appetite for v0.x in the path of 100 % of the traffic): Higress with GIE. CNCF, proven at scale, Apache 2.0, and migratable to the standard EPP.

Control layer (optional, behind): LiteLLM (MIT) if you need a single multi-provider point with virtual keys, spend tracking and fallbacks to external models. It goes behind the data gateway, not in its place, and you accept that SSO, audit and RBAC are solved with the cluster’s own OIDC and auditing (as we set out in the criterion). If your front end is only on-premise vLLM with no external providers, this layer is probably unnecessary.

What I would not recommend here: Kong (its AI features are gated, which contradicts the licence criterion); and starting with LiteLLM as the main data gateway in a stack that will be multi-replica and K8s-native, since its Python ceiling and its position outside the Gateway API model turn it into debt the day you scale.

Applied to our infrastructure: RKE2 + vLLM + 4×H100

The concrete deployment on the generic reference cluster:

# The InferencePool groups the replicas of a model (GIE resource)
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata: { name: llama-70b-pool }
spec:
 selector: { app: vllm-llama70b }
 extensionRef: { name: llama-70b-epp }  # the Endpoint Picker
---
# The standard Gateway API route points at the pool
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: llama-70b }
spec:
 parentRefs: [ { name: ai-gateway } ]
 rules:
 - backendRefs:
 - group: inference.networking.x-k8s.io
 kind: InferencePool
 name: llama-70b-pool

Three notes on how this fits the blog series:

It lives in layer 1 of the seven-layer stack, in front of the replicas pinned by the RKE2 resource managers. The gateway routes; the kubelet pins; both are resources of the same GitOps cluster.
The EPP reads the prefix cache state of each replica, which connects directly with the engineering of prefix cache hit rate: the gateway is what turns that affinity into actual routing.
The gateway’s gen_ai.* tracing feeds the OTel observability pipeline and lands in Langfuse, closing the loop of the Observe layer.

Traps and things that are not what they seem

“It is open source” without reading what falls under Enterprise. The mistake this post tries to avoid: Kong and LiteLLM are OSS, but the features that justify choosing them for AI (AI plugins in Kong; SSO/audit/RBAC/guardrails in LiteLLM) are partly or wholly gated. “Open source” is the wrong question; “is what I need permissive?” is the right one.

“OpenAI-compatible” ≠ “inference-aware”. Almost all of them expose the OpenAI API. That does not mean they understand the KV cache state, the queue of each replica or the LoRA adapters. API compatibility is table stakes; intelligent routing (EPP) is a different league. Do not confuse “speaks OpenAI” with “routes well”.

Choosing by the RPS benchmark. Bifrost wins on raw overhead, but the bottleneck of an on-premise vLLM fleet is rarely the proxy: it is the GPUs. Optimising the gateway for 5,000 RPS when your replicas serve 300 req/s is solving the problem you do not have. The latency the gateway adds matters; its peak throughput almost never does, in this context.

Phone-home telemetry in a sovereign deployment. Several gateways send telemetry home by default. In an ENS or sovereignty context, that is an audit finding, not a detail. Verify and disable the phone-home before putting the gateway in the path of 100 % of the traffic; document it for the ENS/42001/EU AI Act controls file.

Betting on a v0.x in the critical path with no rollback plan. Envoy AI Gateway v0.5 is promising, but it sits in the path of 100 % of the traffic. If you choose it, have the rollback rehearsed and follow the changelog: v0.x releases break things. The GIE standard mitigates this (you can change implementation), but the specific implementation has to be operated with a safety net.

Confusing the data gateway with the control plane. LiteLLM in front of everything seems to simplify, but it mixes two roles: routing data traffic (where you want K8s-native + inference-aware) and managing multi-provider keys and spend (where LiteLLM shines). Separate them: a GIE-native data gateway, with LiteLLM as a control plane behind it if needed.

Conclusion

The gateway is the piece with the highest cost of reversal in the serving layer, and that forces you to invert the intuitive order of criteria. First the licence, not “is it OSS?” but “is what I need permissive or is it gated?”, which already rules Kong out for OSS AI and puts an asterisk on LiteLLM for its Enterprise governance. Then the fit, where the Gateway API Inference Extension turns inference-aware routing into an Apache 2.0 standard you can couple to without marrying a product. And only after that maturity, docs and features, where honesty forces you to admit that the cutting-edge AI-native option (Envoy AI Gateway v0.5) and the battle-tested-today one (Higress) are still not the same thing. For an RKE2 + vLLM stack with a K8s-native priority, the answer is not a product but a model, GIE/EPP, implemented with Envoy AI Gateway if you look at the trajectory or with Higress if you look at maturity, and LiteLLM as an optional control plane behind. The decision that ages well is the one that couples you to a standard, not to a vendor; in gateways, in June 2026, that standard finally has a name.

References

Kubernetes, Introducing Gateway API Inference Extension (blog oficial, jun 2025): https://kubernetes.io/blog/2025/06/05/introducing-gateway-api-inference-extension/.
Gateway API Inference Extension, doc del proyecto (Endpoint Picker, InferencePool): https://gateway-api-inference-extension.sigs.k8s.io/.
Envoy AI Gateway, sitio y release notes (v0.x, GIE integration): https://aigateway.envoyproxy.io/.
LiteLLM, Enterprise Features (qué está gated): https://docs.litellm.ai/docs/proxy/enterprise · licencia MIT: https://github.com/BerriAI/litellm/blob/main/LICENSE.
Kong, Announcing Kong’s Open Source AI Gateway y matriz de plugins Enterprise: https://konghq.com/blog/product-releases/announcing-kong-ai-gateway.
Higress (Apache 2.0, CNCF, AI gateway): https://higress.cn/en/.
Apache APISIX, APISIX vs Kong (cobertura de IA OSS): https://apisix.apache.org/learning-center/apisix-vs-kong/.
vLLM, Semantic Router v0.1 Iris y Production Stack (routing específico de vLLM): https://blog.vllm.ai/2026/01/05/vllm-sr-iris.html.

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

Tue, 02 Jun 2026 03:00:00 +0200

This post is the natural continuation of Canary, blue-green and shadow for LLM models. There the promotion mechanics dumped all the traffic-splitting complexity into a box we called “LoadBalancer”. The description was operational, good enough to understand the choreography, but structurally vague: what actually does that splitting is an L7 inference router with LLM awareness, a piece of the stack in its own right (layer 1 of the seven layers) that deserves its own post.

TL;DR

In the previous post on canary we called LoadBalancer the piece that splits traffic between the stable v1 and candidate v2 pools. The description was enough to understand the flow, but technically it was blurry: neither an L4 LoadBalancer (kube-proxy, MetalLB, IPVS) nor a generic L7 HTTP LoadBalancer (NGINX or HAProxy with no extension) knows what a model is, what a version is, how many tokens a request costs, what prefix the prompt has or which replica has which KV cache hot. The right piece is an LLM inference router: an L7 proxy with explicit knowledge of the domain. It combines four functions: model catalogue (resolving model=llama-70b@v2 → service.namespace:port), traffic splitting (applying the canary weight with a deterministic hash or deliberate stickiness for A/B), cross-cutting policy (OIDC auth, per-tenant rate limit and quota, pre-prompt PII redaction, lightweight inline guardrails, gen_ai.* tracing propagation) and failover/degradation (if v2 falls over, redirect to v1; if the whole cluster is saturated, return 503 with Retry-After instead of queueing forever). The non-obvious piece that justifies its technical existence beyond the operational one is prefix-aware routing: the router decides which fleet replica each request goes to based on the prompt prefix, so that a RAG system with the same system prompt plus the same block of retrieved documents systematically hits the prefix cache (RadixAttention in SGLang, PrefixCaching in vLLM, KV reuse in TensorRT-LLM) of the same replica, multiplying the hit rate from 5–15 % (blind round-robin) to 60–85 % (prefix affinity). The concrete pieces in May 2026 are LiteLLM Proxy (the simplest option, OpenAI-compatible, declarative YAML catalogue), vLLM Production Stack router (specific to vLLM fleets, aware of the KV cache and the prefix), Envoy AI Gateway (LLM-aware Envoy filters, integrable with Istio), Kong AI Gateway (enterprise alternative with a plugin ecosystem), KGateway (CNCF, still incubating) and the NVIDIA Dynamo router (production-grade, aware of disaggregated prefill/decode serving). In the seven-layer stack it lives in layer 1 (gateway); in the five maturity levels it appears from level 3 onwards; in the seven-phase deployment cycle it is the last piece that F6 closes. This post includes a minimal manifest applicable to a generic 4×H100 SXM cluster.

You are here: DEPLOY (layer 1 of the stack)

The background: what the canary post called a “LoadBalancer”

In Canary, blue-green and shadow for LLM models we described the flow like this: “the LoadBalancer progressively splits traffic following a schedule: 1 % → 5 % → 25 % → 100 %”. It was a correct operational description, and the reader understood the choreography without needing more. But technically it left unnamed a piece that deserves explicit treatment, because neither of the two usual senses of “LoadBalancer” does what that paragraph assumed:

An L4 LoadBalancer (kube-proxy with iptables/IPVS, MetalLB, F5 BIG-IP in TCP mode) splits IP packets without looking inside the payload. It does not know which model is being requested, nor which version, nor how many tokens it carries, nor whether the client has quota. It cannot apply the canary weight “for model X version 2”: to it, every packet heading for the vllm-llama70b VIP is indistinguishable.
A generic L7 HTTP LoadBalancer (NGINX or HAProxy in HTTP mode with no extension, a ClusterIP Service with multiple backends) does split by URL and can route by header, but it does not understand the OpenAI-compatible body of the request. It does not know that {"model": "llama-70b", "messages": [...]} carries the routing key in the model field; it does not count tokens; it does not apply policies over LLM structures; it does not do prefix-aware routing because that requires parsing messages and hashing the common prefix.

The piece the canary post assumed was doing this work is an L7 inference router with LLM awareness. A layer in its own right, with its own configuration, its own CI/CD, its own metrics and its own pitfalls. This post names it and takes it apart.

The analogy: the switchboard and triage of a hospital with multiple specialities

A large hospital receives patients who arrive at A&E through different doors and who need different specialities: trauma, cardiology, paediatrics, oncology. There are three possible models of “front door”.

A single door with no triage. All patients wait in the same room and are sent in arrival order to the first free doctor, whatever their speciality. It works in a village surgery with a single general practitioner. When there are 200 patients a day and 12 specialities, it falls into dysfunction fast: the cardiologist treats sprains, the paediatrician treats heart attacks, specialised resources are wasted. This is the equivalent of the L4 LoadBalancer, which distributes bodies without understanding what they bring.

A door with a receptionist who asks about the symptom. Someone at the front desk asks “what’s wrong?” and directs the patient to the right corridor. The cardiologist sees only cardiology, the paediatrician only children. Better, but the receptionist is slow, does not gauge urgency and does not know the state of the rooms: they may send you to the cardiologist in corridor A when the one in B is free. This is the equivalent of a generic L7 HTTP proxy with path-based routing, which distributes by category but with no information about internal state.

Professional triage with full awareness. A trained triage nurse who knows the catalogue of specialities, knows which bay is busy and which is free, remembers the recurring patient whose file is already open in the system (and sends them to the same doctor for continuity), applies cross-cutting policy (checks insurance cover, records allergens, redirects to paediatric A&E if the patient is a minor) and, if the cardiology room goes down because the electrocardiograph has broken, redirects to the hospital on the other side of the city. This is the piece a large hospital needs. In LLM terms it is called an inference router.

The analogy holds down to the last detail, including the “file already open” one: the patient who returns to the same doctor is exactly the client whose prompt shares a prefix with the one from five minutes ago. If the router sends it to the same replica, that replica still has the KV cache hot and the request hits the prefix cache. If it sends it to a different replica because that one was “next in round-robin”, the KV cache has to be rebuilt from scratch and TTFT doubles. The triage nurse knows this. The blind LoadBalancer does not.

The four functions of the inference router

Function 1 — Model catalogue

The router keeps a declarative catalogue mapping model identity to a concrete deployment:

models:
 - name: "llama-70b" # stable alias
 version: "v2" # canary version
 weight: 5 # 5% of the traffic
 endpoint: "vllm-llama70b-v2.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: canary
 - name: "llama-70b"
 version: "v1"
 weight: 95
 endpoint: "vllm-llama70b-v1.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: stable
 - name: "embedding-multilingual"
 version: "v1"
 weight: 100
 endpoint: "tei-bge-m3.inference.svc.cluster.local:8080"
 capabilities: [embeddings]
 lifecycle: stable

The client sends {"model": "llama-70b", "messages": [...]} without knowing there are two replica pools behind it. The router resolves. If tomorrow you migrate from vLLM to SGLang for a specific version, the client never notices; you change the endpoint in the catalogue and that is it.

What this decoupling buys you is the freedom to move topology without breaking clients. What it costs is keeping the naming convention disciplined (llama-70b is always the stable alias; llama-70b@v2 is the specific version for canary). Without that discipline, the aliases get dirty with llama-70b-prod-fixed-real-final-v3 and the catalogue stops being navigable within a few weeks.

Function 2 — Traffic splitting

The splits from the canary post (1 % → 5 % → 25 % → 100 %) materialise here, not in the inference engine. The router computes a deterministic hash of the request_id (or of the user_id, if you want stickiness) and maps it to the weight range in the catalogue. For a weight of [v1: 95, v2: 5], 5 % of the hash space falls in v2 and 95 % in v1.

Three design decisions that matter:

Hash by random request_id = independent sampling. Each request is an independent observation of the v1 vs v2 distribution. It is the right setting for statistically comparable canaries.
Hash by user_id = per-user stickiness. The same client always sees the same pool. Useful for A/B testing with persisted conversational memory, but it breaks the statistical comparability of the canary because user populations are not symmetric, a pitfall explained in the previous post.
Hash by tenant_id = hard partitioning. Tenant A goes to v1, tenant B to v2. This is the pattern for clients with different SLAs, or for validating v2 in an internal tenant before exposing it to external clients.

Function 3 — Cross-cutting policy

Once, across all models, the router applies:

Auth: OIDC with JWT tokens validated against Keycloak / Authentik. Authorization: Bearer ... headers translated into tenant_id and roles.
Rate limit: token bucket per tenant (X req/min) or per model (Y req/min for llama-70b because it is expensive).
Quota: monthly quota of tokens consumed per tenant. The router counts gen_ai.usage.input_tokens + gen_ai.usage.output_tokens and rejects with 429 Quota exceeded when it runs out.
Pre-prompt PII redaction: Presidio or Llama Guard inline before the prompt touches the model. What the model does not see is not trained on, not logged, not leaked.
Lightweight inline guardrails: PromptGuard 2, Llama Guard 4, Granite Guardian, the ones covered in Guardrails and safety in LLMs, run in the router because their latency (30–150 ms) fits within the TTFT budget.
gen_ai.* tracing propagation: the router starts the parent span, propagates traceparent to the engine and emits the gen_ai.system, gen_ai.request.model and gen_ai.request.version attributes that OTel GenAI tracing consumes.
Semantic cache: for prompts that repeat exactly or with high semantic similarity (embedding cosine > 0.97 against the previous cache), it returns the cached answer without touching the engine. Typical saving in RAG with frequently asked questions: 20–40 % of requests.

Function 4 — Failover and degradation

The router knows the health state of each endpoint (active health probes every 5–15 s, recent TTFT latency) and decides:

If v2 returns persistent 5xx or does not respond, the circuit breaker opens: the router redirects the traffic that was going to v2 towards v1 until the probes go green again. This is automatic canary rollback in its simplest form.
If the whole cluster is saturated (all replicas reporting num_requests_waiting > N for T seconds), the router returns 503 Service Unavailable with Retry-After: 30 instead of queueing forever. It is better to tell the client “come back in 30 seconds” than to keep them waiting four minutes and then time out.
If there is multi-region or multi-cluster, cross-cluster failover via DNS or L7: the primary region goes down and the secondary router takes over.

The non-obvious piece: prefix-aware routing

This is the function a conventional LoadBalancer cannot do and the one that justifies an LLM-specific router beyond the four generic ones.

The KV cache in vLLM, SGLang and TensorRT-LLM can reuse common prefixes across requests, see KV cache. Specifically:

vLLM with --enable-prefix-caching: it detects that the current request shares a prefix (of a length that is a multiple of the block size, 16 tokens by default) with a previous request whose pages are still in HBM, and reuses those pages instead of reprocessing them.
SGLang with RadixAttention: it structures the cache as a radix tree indexed by tokens; each request hits the common path of the tree and only computes the tail.
TensorRT-LLM: a similar feature, called KV cache reuse.

The prefix cache hit rate is the key metric: every token that hits is a token that is not processed in prefill, reducing TTFT in direct proportion. For a typical RAG system, with a 400-token system prompt plus 2,000 tokens of retrieved documents plus a 50-token user question, the common prefix (system_prompt + docs) is 2,400 of the 2,450 total tokens. If the cache hits, prefill only processes 50 tokens instead of 2,450: TTFT falls to roughly a twentieth.

But the cache lives per replica, not globally. If two requests with the same 2,400-token prefix land on different replicas, both do the full prefill: the first one’s cache is no use to the second. The second pays the full cost.

With blind round-robin (any conventional LB), requests are spread uniformly across N replicas. For a cluster of 4 replicas and 1,000 requests with the same system_prompt + docs, each replica receives ~250 requests, but all 4 do their own “first prefill” and the following 249 benefit within their replica. The global hit rate is decent but not optimal. For traffic with many different system prompts and little intra-prefix repetition, the hit rate hovers around 5–15 %.

With prefix-aware routing, the router computes a hash of the prompt prefix (the first N tokens, or the system_prompt declared in messages[0]) and keeps an affinity table of hash → replica. All requests with the same prefix land on the same replica. The first pays the full prefill; the following 999 hit the cache. Global hit rate: 60–85 %.

The cost of implementing it: the router has to parse the request body (not just the HTTP header), apply a lightweight tokeniser or a byte-based hash, and maintain an LRU/consistent-hash affinity table that rebalances when a replica comes in or out. That is server work, not generic proxy work. vLLM Production Stack router implements it natively. NVIDIA Dynamo does too. LiteLLM has a beta in its enterprise version. Envoy AI Gateway is adding it as an experimental filter.

Correction (September 2026). The previous sentence about LiteLLM is false: prefix-aware routing does not exist in any version, open or paid, verified against the code of 1.102.0. The detail, what LiteLLM does have, and the real state of the field are in Prefix routing: what LiteLLM does not do.

The operational difference for a production RAG system: with prefix-aware routing, the same cluster serves 2–4× more requests without adding GPUs, simply because prefill disappears in most cases.

Token-aware load balancing

The second non-obvious piece. Classic round-robin splits by number of requests; but a 50-token prompt and an 8,000-token one cost radically different amounts (a factor of ~160× in prefill). Splitting equally by count severely unbalances the real load.

Token-aware load balancing adds up the expected prefill tokens (prompt length) and expected decode tokens (the client’s max_tokens) per active replica, and sends the new request to the replica with the lowest accumulated load. It is what both vLLM Production Stack and NVIDIA Dynamo implement as the default strategy when it is enabled.

The metric feeding the calculation is, once again, vllm:num_requests_running and vllm:gpu_cache_usage_perc, see GPU observability for LLM inference, ideally complemented with an estimator of the incoming prompt’s tokens. Mature routers use tiktoken or the model’s real tokeniser to count prompt tokens before choosing a replica.

Comparison of concrete pieces (May 2026)

Piece	LLM awareness	Prefix-aware	Token-aware LB	Multi-model	Semantic cache	Plug & play
LiteLLM Proxy	High	Beta (enterprise)	Yes	Excellent	Yes (Redis)	Very high
vLLM Production Stack router	vLLM-specific	Yes, native	Yes	vLLM only	No (external)	Medium
NVIDIA Dynamo router	High + disagg-aware	Yes	Yes	vLLM/SGLang/TRT-LLM	No (external)	Low
Envoy AI Gateway	Medium (filters)	Experimental	Yes	Yes	Via filter	Medium
Kong AI Gateway	Medium (plugins)	No	Yes	Yes	Yes (plugin)	Medium
KGateway	Medium	Roadmap	Yes	Yes	Roadmap	Low (CNCF incubating)
NGINX + custom Lua	Manual	No	Manual	Manual	No	Low (build it yourself)

LiteLLM Proxy is the default option to start with. OpenAI-compatible, simple YAML, supports the commercial providers plus any self-hosted OpenAI-compatible backend. The OSS version covers the four basic functions and semantic cache; prefix-aware routing and the enterprise version add advanced multi-tenancy.

vLLM Production Stack router is the right option if the fleet is 100 % vLLM. Aware of the KV cache, the prefix and the LoRA loaded per replica. It integrates better with native vLLM metrics.

NVIDIA Dynamo router is the most complete production-grade option, especially if you operate disaggregated serving (separate prefill workers and decode workers). It requires an NVIDIA-aligned stack.

Envoy AI Gateway and Kong AI Gateway are the options if the organisation already has Envoy or Kong as its corporate gateway and wants to extend it with LLM awareness without introducing another new piece.

Minimal manifest: LiteLLM Proxy on a generic cluster

apiVersion: v1
kind: ConfigMap
metadata: { name: litellm-config, namespace: inference }
data:
 config.yaml: |
 model_list:
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v1.inference.svc:8000/v1
 weight: 95
 model_info:
 version: v1
 lifecycle: stable
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v2.inference.svc:8000/v1
 weight: 5
 model_info:
 version: v2
 lifecycle: canary
 - model_name: embedding-multilingual
 litellm_params:
 model: openai/bge-m3
 api_base: http://tei-bge-m3.inference.svc:8080
 router_settings:
 routing_strategy: least-busy # basic token-aware
 num_retries: 1
 timeout: 60
 general_settings:
 master_key: "os.environ/LITELLM_MASTER_KEY"
 database_url: "os.environ/DATABASE_URL"
 litellm_settings:
 cache: true
 cache_params:
 type: redis
 host: redis.inference.svc
 port: 6379
 similarity_threshold: 0.97
 success_callback: ["langfuse"]
 failure_callback: ["langfuse"]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: litellm-router, namespace: inference }
spec:
 replicas: 3
 selector: { matchLabels: { app: litellm } }
 template:
 metadata: { labels: { app: litellm } }
 spec:
 containers:
 - name: litellm
 image: ghcr.io/berriai/litellm:v1.55.0
 args: ["--config=/config/config.yaml", "--port=4000", "--num_workers=4"]
 ports: [{ containerPort: 4000, name: http }, { containerPort: 4000, name: metrics }]
 env:
 - { name: LITELLM_MASTER_KEY, valueFrom: { secretKeyRef: { name: litellm-secret, key: master_key } } }
 - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: litellm-secret, key: db_url } } }
 - { name: LANGFUSE_PUBLIC_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: public } } }
 - { name: LANGFUSE_SECRET_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: secret } } }
 volumeMounts: [{ name: config, mountPath: /config }]
 readinessProbe: { httpGet: { path: /health, port: 4000 } }
 volumes: [{ name: config, configMap: { name: litellm-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: litellm-router, namespace: inference }
spec:
 selector: { app: litellm }
 ports: [{ name: http, port: 80, targetPort: 4000 }]
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: litellm-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: litellm } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s

The end client points at litellm-router.inference.svc:80/v1/chat/completions, sets model=llama-70b, and the router decides on each request whether it goes to v1 (95 %) or v2 (5 %), applies the rate limit, looks in the semantic cache, propagates tracing to Langfuse, and translates from OpenAI-compatible to the OpenAI-compatible API of the destination vLLM. Three router replicas for HA and so the gateway itself can scale horizontally with KEDA if needed, see LLM autoscaling on Kubernetes.

Four operational pitfalls

Pitfall 1 — the router becomes a SPoF if it is not replicated. Three or more replicas of the router itself, behind a LoadBalancer Service (this one genuinely L4) with healthchecks. A single router replica means every configuration deploy closes the whole service for a few seconds.

Pitfall 2 — the router’s latency adds to the model’s. Each function adds milliseconds: body parsing (5–10 ms), JWT auth (2–5 ms), rate limit (1–2 ms), PII redaction with Presidio (20–80 ms), inline guardrails with Llama Guard (50–150 ms), prefix hash (5–10 ms), token counting with a tokeniser (10–30 ms). In total, 100–300 ms of overhead before touching the engine. If the model’s TTFT is 400 ms and the router’s is 200 ms, the client sees 600 ms, so it is worth measuring what each function costs and disabling the non-critical ones in the low-latency path.

Pitfall 3 — the catalogue drifts from the real state of the cluster. The router believes vllm-llama70b-v2 exists because it is in its YAML, but the deployment was withdrawn three days ago and nobody updated the config. The router returns 502 on 5 % of the traffic. Fix: validate the catalogue against kubectl get svc in CI; no endpoint in the catalogue may point at a non-existent Service. Or better: the router discovers the available endpoints dynamically via a label selector (app=vllm,model=llama-70b) and applies the catalogue weights over the ones that are alive.

Pitfall 4 — semantic cache with an outdated embedding. The semantic cache compares the new prompt’s embedding against the embeddings of cached prompts. If you update the embedding model (see RAG corpus curation), the distances are computed in a different space and the cache stops working correctly (false hits or false misses). Policy: the cache is invalidated when the embedding model changes; generations are never mixed.

Fit within the stack and maturity

In the seven-layer stack, the router is layer 1: the front door that precedes the inference engine (layer 2), the KV cache + PagedAttention (layer 3) and the rest. It is the only piece that sees all the traffic from the outside; any policy not applied here gets duplicated N times in the engines.

In the five maturity levels, the router appears from level 3 (MANAGED) onwards: without OIDC + RBAC + cert-manager + default-deny NetworkPolicy, the router has nobody to authenticate and nobody to apply quotas to; before level 3, what you should build is a minimal proxy with no pretension of a catalogue. Platforms that try to have a polished router at level 1 end up with a large YAML nobody maintains.

In the seven deployment phases, the router is what closes F6: the last atomic step that puts the cluster into production. Without a router, F6 does not finish, because the catalogue, the quotas, the canaries and the failovers are a necessary condition for opening production traffic.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB, the inference router consumes modest resources: 3 replicas of the router pod (2 CPU cores, 4 GiB of memory each) are enough to sustain thousands of RPS because its work is light (parsing, hashing, routing, no inference). The router lives on non-GPU nodes of the cluster (control plane nodes or general workload nodes), and never consumes nvidia.com/gpu.

Traffic volume that a LiteLLM with 3 replicas and 4 workers each will sustain: 2,000–5,000 RPS routing to a vLLM backend, with 80–150 ms of overhead on the full path (auth + rate limit + cache check + propagation). If more is needed, scaling the router with KEDA on litellm:requests_per_second is trivial.

For larger clusters (16+ GPU nodes), consider the vLLM Production Stack router or the NVIDIA Dynamo router, which are more complex but squeeze out the prefix-aware routing and token-aware LB that LiteLLM OSS does not cover. For multi-region clusters, Envoy AI Gateway with Istio Service Mesh is the standard choice.

What we have not covered (upcoming articles)

Deep comparison of LiteLLM vs vLLM PStack vs Dynamo, with prefix-aware benchmarks on a real on-premise cluster.
Semantic cache with Redis Stack + RedisVL: hit rate, false positives, TTL policy.
Multi-region routing: how the router decides between the DC1 and DC2 clusters based on latency, health and load.
AI Gateway specific features: token-bucket cost-based rate limiting (penalising long prompts), guardrails policy engine in the router.
Migration path: how to introduce a router into a cluster that already has clients pointing straight at the vLLM service, with no downtime.

References

LiteLLM project — litellm.ai (documentación de Proxy, routing strategies, semantic cache).
vLLM Production Stack — github.com/vllm-project/production-stack (router con prefix-aware nativo).
NVIDIA Dynamo — developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/ (router production-grade con disaggregated-aware).
Envoy AI Gateway — gateway.envoyproxy.io/docs/tasks/ai-gateway/ (proyecto en gestación dentro de Envoy).
Kong AI Gateway — konghq.com/products/kong-ai-gateway (proxy enterprise con plugin LLM).
KGateway — kgateway.dev (alternativa CNCF en gestación).
Zheng et al. — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention y prefix caching.
vLLM project — Automatic Prefix Caching (docs.vllm.ai/en/latest/features/automatic_prefix_caching.html).
Patel et al. — SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024) — base teórica del routing prefill/decode aware.

Router on lo0 — Tech Blog

Choosing the switchboard: which OSS gateway to put in front, and why the licence decides before the features

TL;DR

The principle that reorders the criteria: the gateway is a marriage, not a date

The field of candidates (June 2026)

Filter 1 — Licence: where the small print is

LiteLLM: what is gated and its OSS equivalent

Filter 2 — Fit: Kubernetes citizen or tenant?

Filter 3 — Maturity and documentation: the uncomfortable honesty

Decision matrix

The recommendation

Applied to our infrastructure: RKE2 + vLLM + 4×H100

Traps and things that are not what they seem

Conclusion

See also

References

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

TL;DR

You are here: DEPLOY (layer 1 of the stack)

The background: what the canary post called a “LoadBalancer”

The analogy: the switchboard and triage of a hospital with multiple specialities

The four functions of the inference router

Function 1 — Model catalogue

Function 2 — Traffic splitting

Function 3 — Cross-cutting policy

Function 4 — Failover and degradation

The non-obvious piece: prefix-aware routing

Token-aware load balancing

Comparison of concrete pieces (May 2026)

Minimal manifest: LiteLLM Proxy on a generic cluster

Four operational pitfalls

Fit within the stack and maturity

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References