Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

· By José Manuel Román
Contents

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs \( C \) euros per hour and your service is idle \( H \) hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

See also

Sources