Keda on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

Mon, 31 Aug 2026 10:00:00 +0200

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs $ C $ euros per hour and your service is idle $ H $ hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

Sources

CNCF, Cloud Native Computing Foundation announces Knative’s graduation — https://www.cncf.io/announcements/2025/10/08/cloud-native-computing-foundation-announces-knatives-graduation/
Knative Docs, Serving architecture — https://knative.dev/docs/serving/architecture/
Knative Docs, Request flow — https://knative.dev/docs/serving/request-flow/
Knative Docs, Configuring KPA-specific autoscaling — https://knative.dev/docs/serving/autoscaling/kpa-specific/
Knative Docs, Configuring concurrency — https://knative.dev/docs/serving/autoscaling/concurrency/
Knative Docs, Configuring scale-to-zero — https://knative.dev/docs/serving/autoscaling/scale-to-zero/
Knative Blog, Managing Gateway API ingress with the Knative Operator — https://knative.dev/blog/articles/gateway-api-ingress-with-knative-operator/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Autoscaling with Kubernetes HPA (Serverless vs Standard) — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/hpa-autoscaler
KServe Docs, Autoscaler for generative inference (KEDA) — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
KServe GitHub, docs/samples/autoscaling (InferenceService minReplicas 0) — https://github.com/kserve/kserve/blob/master/docs/samples/autoscaling/README.md
Wang et al. (MLSys 2026), Breaking the ice: analyzing cold start latency in vLLM — https://arxiv.org/abs/2606.07362
NVIDIA Developer Blog, Reducing cold start latency for LLM inference with NVIDIA Run:ai Model Streamer — https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer
Microsoft Azure SDK Blog, Eliminate LLM cold starts: load models up to 6x faster — https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
Hugging Face Blog, Llama 3.1 (VRAM and precision tables) — https://huggingface.co/blog/llama31
CloudZero, H100 GPU cost 2026 — https://www.cloudzero.com/blog/h100-gpu-cost/

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

Keda on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

TL;DR

The analogy: the restaurant that switches off the kitchen

How Knative scales

KPA versus HPA

Why an LLM cold start is not starting a pod

The techniques that shorten the wait

Knative and KServe: the two modes

The economics

Operational traps and honest scepticism

For an inference factory

See also

Sources

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

TL;DR

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

Why HPA on CPU is no use

The four viable signals

The wiring: KEDA as the Prometheus adapter

The big operational problem: cold start

The five levers

When to scale nodes, not just pods

Three pitfalls specific to LLM scale-down

Full example manifest

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References