Nvme on lo0 — Tech Blog

Storage in the AI era (1/4): the state of the art

Mon, 22 Jun 2026 12:00:00 +0200

Public conversation about AI infrastructure almost always revolves around the GPU: how many FLOPs, how much HBM, how many watts. But anyone who designs or runs an AI factory knows the bottleneck has moved. A Blackwell GPU sitting idle waiting for data costs exactly the same as a Blackwell GPU doing work; what makes the difference is the subsystem feeding it. This first article in the series maps the state of the art in AI storage as of mid-2026: the memory and storage hierarchy, the media technologies (NAND, NVMe, CXL), the parallel file systems that dominate GPU clusters and the new inference stack that has turned the SSD into a component of the memory hierarchy.

The three articles that follow go deeper into the three properties that matter to an architect once that storage reaches production: performance, security and availability.

The underlying problem: the memory wall

A modern GPU is, above all, a machine for moving data. Compute itself is almost never the limit; the limit is the bandwidth feeding the functional units. The hierarchy that runs from GPU registers down to NVMe spans six orders of magnitude in latency and several in bandwidth. Registers are on the order of a million times faster than an NVMe access, and in between sit on-chip SRAM, HBM, host DRAM and, finally, persistent storage.

The practical consequence is well known: in many training workloads GPUs run at 30-50 % of their theoretical FLOPs utilisation because they spend a significant fraction of their time waiting for data. This is the memory wall. Its most useful quantitative formulation is arithmetic intensity, the ratio of operations to bytes moved:

$$I = \frac{\text{FLOPs}}{\text{bytes}}$$

When a workload’s intensity falls below the hardware’s break-even point (the ridge point of the roofline model), performance is limited by bandwidth rather than compute. The industry’s race towards HBM, towards NVLink and, at the layer that concerns us, towards 28 GB/s NVMe and 800 Gb/s networks, is at bottom a war against that wall.

The problem gets worse because compute has grown much faster than memory bandwidth. Each GPU generation multiplies FLOPs more generously than the TB/s of memory that feed them, so the ridge point shifts towards ever higher arithmetic intensities: more workloads fall on the bandwidth-limited side. This is the structural reason storage and memory have moved to the foreground. It is not that storage has improved less (it has improved enormously), but that the GPU demands more of it every year. An architect who designs with FLOPs alone in mind will build systems that spend half their time waiting.

It pays to keep the whole pyramid in view when we talk about “storage” in AI, because design decisions at one level constrain the others:

Level	Technology	Typical bandwidth	Latency	Capacity per device
On-package	HBM3e / HBM4	1.2-3.3 TB/s per stack	ns	36-64 GB
Host memory	DDR5 / CXL	0.3-0.5 TB/s	~100 ns	TB
Hot flash	NVMe Gen5/Gen6	14-28 GB/s	10-100 µs	4-256 TB
Capacity	Nearline QLC / HDD	1-7 GB/s	ms	30-245 TB
Archive	Object / tape	variable	s	EB (aggregate)

The state of the art in 2026 consists, to a large extent, in how the boundaries between these levels blur: HBM spills over into DRAM, DRAM into CXL, and the inference KV-cache into NVMe. Storage has stopped being the basement of the hierarchy and has become part of it.

HBM4: the ceiling rises, but it stays expensive

At the top sits high-bandwidth memory stacked next to the GPU. In April 2025 JEDEC published the HBM4 standard (JESD270-4), which doubles the interface to 2048 bits per stack against HBM3e’s 1024 bits, reaches up to 8 Gb/s per pin and lands around 2 TB/s typical with peaks up to 3.3 TB/s per stack. It supports configurations of 4 to 16 layers with 24 or 32 Gb dies, up to 64 GB per cube. HBM3e, the generation that dominates accelerators in production in mid-2026, sits near 1.2 TB/s and a 36 GB maximum.

The figure that matters to an architect is not peak bandwidth but scarcity. Micron has essentially sold out its entire HBM production through 2026, and capacity expansion is not expected before late 2027-2028. That constraint is what pushes the load downwards through the hierarchy: if you cannot add HBM, you have to learn to spill gracefully into DRAM, CXL and flash. Much of the recent innovation comes from there.

NAND and SSD: capacity explodes, the interface widens

The flash layer is where AI’s pressure on the market shows most. Two vectors at once: capacity and interface.

On capacity, 2025-2026 has been the year of the race to 245 TB. The Solidigm D5-P5336 reached 122 TB (PCIe 4.0) and became the highest-capacity SSD available, on sale for around 12,400 USD; Solidigm has confirmed models of 245 TB or more for late 2026. Micron introduced the 6600 ION with 245 TB in E3.L form factor. By mid-2026 there are at least eight 245 TB SSDs announced by SanDisk, Samsung, Kioxia, Micron, SK hynix and others, all built on high-density QLC NAND (SK hynix is already producing 321-layer QLC) to satisfy hyperscaler demand. Enterprise QLC is consolidating as a replacement for nearline HDD, whose supply lags well behind demand and whose lead times have stretched.

QLC deserves a qualification that sometimes gets lost in the excitement over capacity: write endurance. QLC NAND stores four bits per cell, which multiplies density but reduces the number of erase cycles it tolerates compared with TLC. For AI datasets written once and read many times, the typical case of a training set, this is irrelevant and QLC is the right choice. For workloads with intensive, constant writes, certain checkpoints or a fast-rotating KV-cache, wear has to be watched and TLC often reserved. Knowing the write pattern of each workload is what allows the right medium to be assigned without overpaying or under-wearing.

On interface, PCIe Gen6 has entered datacenter production. The Micron 9650 was the first mass-produced PCIe Gen6 datacenter SSD (February 2026), with 28 GB/s of throughput, roughly double Gen5. Controllers are keeping pace: the Silicon Motion SM8466 advertises 28 GB/s sequential and 7 million random IOPS; FADU’s Gen6 part (“Lhotse”) targets 28.5 GB/s read, 6.9 million IOPS, NVMe 2.2 and native SR-IOV with 128 virtual functions. The adoption path is clear: Gen6 arrives first in AI, HPC and the datacenter, and only afterwards, if at all, in the consumer PC.

Over the network, NVMe-oF splits into two worlds. NVMe/TCP leads greenfield array-to-host deployments for its operational simplicity, with latencies of 100-200 µs. NVMe over RDMA (RoCEv2, iWARP) remains the gold standard for low latency (10-20 µs) where a lossless fabric already exists, such as the ones AI clusters deploy. Ethernet at 25-100 Gb/s is steadily eroding Fibre Channel’s share of storage traffic.

The choice between TCP and RDMA is not an aesthetic preference. NVMe/TCP runs over any existing Ethernet network and greatly simplifies operations, at the cost of higher latency and of host CPU consumption that, under heavy load, competes with compute. NVMe/RoCE demands a lossless fabric, with priority flow control and congestion control properly tuned, plus capable NICs, but it delivers the latency that latency-sensitive AI workloads need and offloads processing to the NIC. In an AI cluster that already deploys InfiniBand or RoCE for GPU-to-GPU traffic, reusing that fabric for storage is the natural decision; in a mixed enterprise infrastructure, NVMe/TCP usually wins on operability.

CXL: from PowerPoint to production

Compute Express Link spent years as the eternal promise. By early 2026 it has finally reached mainstream adoption in production platforms with CXL 3.0/3.1, aligned with PCIe 6.1. The hardware support is there: Intel Xeon 6 (Granite Rapids and later) includes native CXL 3.0 in the CPU, and AMD EPYC Turin is a target platform. CXL 3.0 multi-host pooling switches are already running at some colocation providers.

The use case that has moved from theory to numbers is memory tiering: keeping hot data in GPU HBM and pushing cold data to a shared CXL DRAM pool. Supply chain vendors (Astera Labs, MemVerge, XConn) report notable gains from offloading the inference KV-cache to CXL memory compared with SSD or RDMA-based solutions. The specific figures deserve caution, since they come largely from the manufacturers themselves, but the direction is real: CXL has become a usable disaggregated memory layer, not a laboratory experiment. Tiering support in the Linux kernel (6.1 onwards) is maturing, although getting value from it still requires NUMA-aware applications.

Parallel file systems: the GPU battlefield

If the media layer is where silicon vendors compete, the parallel file system layer is where platform vendors compete, and it is the one an AI architect touches daily. The market has reshuffled in a few years.

WEKA has probably been the most talked-about case. Its system (WekaFS, now repackaged as NeuralMesh) bypasses the Linux kernel by running as an RTOS in user space, uses RDMA instead of TCP/IP and distributes metadata through consistent hashing. Published figures cite latencies of 100-200 µs against more than 1000 µs for traditional stacks, and metadata performance that doubles Lustre’s. In 2026 it launched its NeuralMesh AI Data Platform on the NVIDIA reference design, with an “Augmented Memory Grid” aimed at context memory for agentic AI.

VAST Data has bet on a different architecture, DASE (Disaggregated Shared-Everything), a single all-flash tier on which it has built what it calls an “AI OS”: a stack with a database, a data engine, InsightEngine for real-time AI pipelines, vector search and RAG, and integrated governance. Azure has adopted its system, a sign of the traction it has gained.

Lustre (DDN) remains the most widespread file system in classic supercomputing, but it is being eroded in the GPU arena by VAST and WEKA. DAOS shines at the top end of traditional HPC but barely appears in commercial AI clusters. IBM Storage Scale (the former Spectrum Scale/GPFS) has repositioned itself with Content Awareness for RAG and support for NVIDIA’s data platform, alongside a refresh of the FlashSystem range.

The common denominator of this whole generation is alignment with NVIDIA. Platforms are certified against DGX SuperPOD and integrate with GPUDirect Storage, which allows direct DMA between storage and GPU memory, skipping the CPU and the bounce buffer. In GB200 NVL72 architectures, a liquid-cooled rack that presents 72 GPUs as a single giant GPU through NVLink, with 13.4 TB of unified GPU memory, GPUDirect Storage certification stops being an ornament and becomes a requirement.

To place the main players side by side, it helps to summarise their architectural bets, which are genuinely different and not mere variations on a single design:

Platform	Architecture	Differentiator	Fit in AI
WEKA NeuralMesh	User-space RTOS, RDMA	Very low latency, metadata distributed by hashing	Training and agentic inference, context memory
VAST DataStore	Disaggregated all-flash DASE	Single exabyte-scale multiprotocol namespace, AI OS	Lakehouse + RAG + inference
IBM Storage Scale	Parallel (ex-GPFS)	Content Awareness for RAG, HPC maturity	Enterprise and HPC with unstructured data
DDN EXAScaler/Lustre	Hardened Lustre	Raw throughput per rack, massive deployments	Supercomputing and AI factories
Pure Storage	All-flash with Purity	Evergreen operations, simplicity	Enterprises that value operations over peak

The choice between them is rarely decided by peak GB/s, but by workload pattern (sequential training against random inference), by the operating model and by the degree of integration required with the NVIDIA stack. A sensible architect first assesses their workload mix and only then compares numbers.

The rack as the unit of compute: GB200, Rubin and BlueField-4

The change in granularity is worth pausing on. For decades the design unit was the server; today, in AI, it is the rack. The GB200 NVL72 integrates 72 Blackwell GPUs and 36 Grace CPUs into a single 130 TB/s NVLink domain, presented to software as one logical GPU with 13.4 TB of unified memory and capable of up to 30 times the real-time inference of a trillion-parameter LLM against the previous generation. At that scale storage does not compete with internal bandwidth (it could not), but it must feed model loading and the inference KV-cache (and, where applicable, training checkpointing) without strangling the 72 GPUs that behave as one.

The roadmap sharpens the trend. The Rubin generation and, in particular, the Rubin CPX, a GPU specialised in long-context inference with more than a million tokens, push still more data towards the memory and storage layers. And the piece that stitches it all together is the DPU: NVIDIA’s BlueField-4 enables an “Inference Context Memory Storage” platform that turns the KV-cache into a pod-level shared flash resource, connected over Spectrum-X Ethernet. The DPU offloads storage and security data movement from the CPU, and is becoming the true storage controller of the AI node. For the architect, this means storage design no longer ends at the array: it includes the NIC, the DPU and the fabric.

The inference stack: when the SSD joins the memory hierarchy

The most important conceptual change of the past two years, and the one that best summarises the state of the art, is this: flash storage has become part of the inference memory hierarchy. The cause is context windows. As models handle hundreds of thousands, or millions, of tokens, the KV-cache generated by the prefill phase overflows HBM. That overflowing KV-cache drops first into host DRAM and then to the NVMe SSD.

NVIDIA has standardised this pattern. Its Dynamo orchestration layer operates across the whole hierarchy, HBM → CPU DRAM → NVMe → network storage, with a KV-cache offload engine and a transfer library (NIXL) for sharing cache between nodes. In January 2026 it standardised KV-cache offload to NVMe SSD, and it has introduced an “Inference Context Memory Storage” platform based on BlueField-4 DPUs that turns the KV-cache into a high-bandwidth shared resource at pod level. Solidigm and other SSD makers have aligned their message around the same idea: long-context inference is, in part, a flash storage problem.

The design consequences are concrete. A 2026 inference server carries between 8 and 32 SSDs, with the figure expected to reach 32 within a couple of years, and a random access pattern, against the sequential pattern and the dozen or so drives of a training server. The ratio of inference to training servers can reach 50 to 1. The centre of gravity of storage investment is shifting towards inference.

Anatomy of storage in an inference factory

It is worth pausing here, because an inference AI factory, a system designed to serve models in production rather than train them, has a storage profile of its own, different from the training cluster that dominates the literature. While training is a batch job that tolerates stoppages and is measured in weeks, inference is a continuous service, latency-sensitive and bound by an SLA: every millisecond and every failure shows up at the client. Its storage is organised into four planes that are best designed separately.

The first plane is the model registry. The weights of the models being served, which may number in the dozens, with versions, quantised variants and LoRA adapters, live in object storage or a shared file system, and are loaded into GPU HBM when a model is activated. The critical requirement here is not sustained throughput but load time: when a new replica has to start or a GPU has to switch models, reading tens or hundreds of GB of weights determines the cold start. A slow registry translates into sluggish autoscaling and GPUs that take their time entering service, exactly when demand rises.

The second plane is the KV-cache hierarchy, the heart of modern inference performance. As we saw, the KV-cache overflows HBM and drops to DRAM, CXL and NVMe. In an inference factory this is not a marginal detail: it is a first-class storage layer, with its own sizing, its random access pattern and critical latency, and a lifetime measured in seconds. NVIDIA’s standardisation of KV-cache offload to NVMe SSD and the ICMS platform on BlueField-4 exist precisely for this: to allow long contexts and cache reuse across requests (and across nodes) without paying the price of recomputation or of adding HBM, which is scarce and expensive.

The third plane is the knowledge store for RAG: the vector databases (Milvus, Weaviate, Pinecone, or S3 Vectors) and the source documents the system retrieves to enrich its answers. The pattern here is read-intensive, with similarity searches over indexes that can run to TB and are best kept on low-latency flash. The availability and freshness of this store determine answer quality as much as the model itself, and updating it, reindexing and taking in new documents, is a constant write flow the design has to account for.

The fourth plane is observability and logging: traces, request and response logs, metrics and, where regulation demands it, the auditable archive of interactions. It is a high-volume append-only write flow that usually ends up in object storage, and one that a GPU-centric design tends to forget about until it saturates something.

The conclusion for anyone building an inference factory is that its storage looks nothing like a rescaled training cluster. It is sized by model load time, by KV-cache capacity and latency, by vector search performance and by telemetry volume, with a cross-cutting imperative of continuous service under SLA. Training, where it exists, is the occasional guest; inference is the permanent resident.

The other end: object, lakehouse and vectors

Not everything is hot flash. The training database and the system of record live in object storage. The novelty of 2025-2026 is convergence with open table formats: Apache Iceberg, Delta Lake and Hudi bring ACID transactions, time travel and schema evolution on top of cheap object storage. AWS has launched S3 Tables (managed Iceberg, with faster queries) and S3 Vectors (native vector storage with savings of up to 90 % for AI workloads). Vector databases, Pinecone, Milvus, Weaviate, integrate with data lakes to feed RAG, and columnar formats such as Parquet, together with Lance for multimodal data, are consolidating as the substrate of the lakehouse.

Computation near the data (computational storage) remains active R&D rather than mass deployment. The corresponding NVMe specifications have been published since late 2023, with eBPF as the execution environment for downloadable functions, but real adoption is still marginal. The idea, running filtering, decompression or search directly on the device so the data never travels to the CPU, is seductive for AI workloads that read enormous datasets at low arithmetic intensity, but the programming model and the lack of tooling standardisation hold deployment back. It is a technology to watch, not yet one to put into production.

The data lifecycle: tiering as an architectural decision

Taking in the whole picture, it is clear that no data lives in a single place for its entire lifecycle. A dataset arrives raw in the object data lake, is transformed and catalogued in an open table format (Iceberg, Delta), is promoted to the hot flash layer of the parallel file system to feed training, generates checkpoints that live for days on NVMe, and finally, together with the resulting models, is archived to cold object storage or tape. Automatic tiering between these layers has gone from a cost optimisation to a first-order architectural decision, especially amid NAND and HBM scarcity.

The practical rule that emerges is to place every byte in the cheapest tier that satisfies its latency and bandwidth requirement at that moment, and to automate the transitions. Hot flash (NVMe Gen5/Gen6) absorbs the training working set and the inference KV-cache; capacity QLC replaces nearline HDD for datasets that are read sequentially; object storage is the durable, versioned system of record; and tape is resurging as an active archive at exabyte scale to fight the cost of flash. Designing those boundaries, and measuring how much data crosses each one, is now part of the AI storage architect’s job, not the administrator’s.

The market context: a supply supercrisis

None of this happens in an economic vacuum. Storage demand from AI servers is 8 to 10 times that of a traditional server, and it is growing above 20 % a year, while global NAND supply grows only 15-17 %. The result, in the words of several analysts, is a memory “supercrisis”: high double-digit contract price rises for DRAM and NAND in 2026, stretched lead times and collateral effects on smartphones and PCs. For an architect, this turns capacity efficiency (QLC, compression, erasure coding, tiering) into a first-order cost lever rather than an implementation detail.

What this changes for the architect

If the state of the art had to be distilled into a handful of practical implications, these would be them. First, storage is designed by workload pattern, not by capacity: a training server and an inference server have opposite requirements, sequential against random, few drives against many, and mixing them in a single template is a common mistake. Second, the limit almost never sits in the medium any more but in the full chain of medium, NVMe, DPU, NIC, fabric and file system; optimising one link while leaving the rest untouched does not move the needle. Third, supply scarcity turns capacity efficiency (QLC, compression, erasure coding, tiering) into a cost lever comparable to buying the drives themselves. And fourth, the boundary between memory and storage has dissolved: thinking in terms of “RAM on one side, disk on the other” no longer describes a modern AI system, where HBM, DRAM, CXL and NVMe form a software-managed continuum.

Takeaways

The state of the art in AI storage in 2026 comes down to one idea: the hierarchy has become continuous. HBM spills into DRAM, DRAM is disaggregated over CXL, the inference KV-cache drops all the way to NVMe, and the parallel file system orchestrates the whole so the GPU never waits. Media technologies (HBM4, 245 TB QLC, NVMe Gen6) raise the ceilings, but supply scarcity turns efficiency into a design constraint. And the software battlefield, WEKA, VAST, IBM, DDN, is fought over a single metric: keeping the GPUs fed.

With this map on the table, the three articles that follow go into the detail of the properties that decide whether a storage architecture works in production.

Sources

FADU, From Training to Inference: FADU at CFMS 2026 — https://blogs.fadu.io/ai-inference-ssd-data-centers/
Avnet, Riding the AI Supercycle: 2026 Memory & Storage Market — https://www.avnet.com/integrated/resources/article/2026-memory-shortage-ai-supercycle/
JEDEC, JESD270-4 HBM4 Standard — https://www.jedec.org/news/pressreleases/jedec%C2%AE-and-industry-leaders-collaborate-release-jesd270-4-hbm4-standard-advancing
Tom’s Hardware, JEDEC finalizes HBM4 memory standard — https://www.tomshardware.com/pc-components/ram/jedec-finalizes-hbm4-memory-standard-with-major-bandwidth-and-efficiency-upgrades
StorageNewsletter, Micron 9650 PCIe Gen6 data center SSDs in mass production — https://www.storagenewsletter.com/2026/02/19/micron-9650-pcie-gen6-data-center-ssds-in-mass-production/
Tom’s Hardware, Solidigm reveals 122TB SSD — https://www.tomshardware.com/pc-components/ssds/solidigm-reveals-122tb-ssd-the-worlds-highest-capacity-drive-for-ai-workloads-d5-p5336-offers-unlimited-write-durability
TechRadar, Solidigm confirms 245TB SSDs before end of 2026 — https://www.techradar.com/pro/solidigm-confirms-245-tb-ssds-set-to-launch-before-end-of-2026
KAD, CXL in 2026: How Memory Pooling is Reshaping Data Centers — https://www.kad8.com/hardware/cxl-in-2026-how-memory-pooling-is-reshaping-data-centers/
Blocks & Files, Parallel filesystem definitions — https://blocksandfiles.com/2025/11/26/parallel-filesystem-definitions-and-powerscale/
NAND Research, WEKA NeuralMesh AIDP & STX integration — https://nand-research.com/weka-neuralmesh-aidp-stx-integration-gtc-2026/
VAST Data, InsightEngine — https://www.vastdata.com/platform/insightengine
NVIDIA, GB200 NVL72 — https://www.nvidia.com/en-us/data-center/gb200-nvl72/
Blocks & Files, Nvidia standardizes GPU cluster KV cache offload to NVMe SSDs — https://blocksandfiles.com/2026/01/06/nvidia-standardizes-gpu-cluster-kv-cache-offload-to-nvme-ssds/
AWS, Building AI-Ready Data Lakes: S3 Tables, Iceberg, S3 Vectors — https://builder.aws.com/content/34vssNUviyG5WPv3IRiGZFKZYkn/building-ai-ready-data-lakes-amazon-s3-tables-apache-iceberg-and-s3-vectors
TrendForce, AI Server Demand to Drive Memory Contract Price Increases 2Q26 — https://www.trendforce.com/presscenter/news/20260331-12995.html

The pantry lift: from disk to HBM, or why the kitchen opens late

Sun, 07 Jun 2026 08:30:00 +0200

This is a trip down to the basement. The under the engine series optimised the hot path, what happens with every token once you are already serving. This post looks at the journey that comes before serving: how the weights climb from disk to HBM. It is the first of a pair on the two things that happen outside the API and that almost nobody times: model loading (this one) and execution on the silicon (the next one).

TL;DR

Before an inference pod generates its first token, it has to push the whole model into HBM. A Llama-70B in FP16 is 140 GB travelling along a path nobody draws: disk → page cache → host buffer → PCIe → HBM. Intuition fails here: HBM is not the bottleneck, it moves 3.35 TB/s and swallows 140 GB in 42 ms; the bottleneck is the supply chain. The Gen5 NVMe disk reads at ~14 GB/s (10 s for 140 GB); PCIe Gen5 copies host→GPU at ~50 GB/s (2.8 s); and the default safetensors loader, which deserialises tensor by tensor and bounces every byte through a CPU buffer, inflates all of that up to 30-60 s. That time is the cold start, and it is the hidden tax paid by autoscaling (scale-from-zero), by canary/blue-green and by disaggregated serving every time a pod is born. There are three families of solution, GPUDirect Storage (direct DMA disk→HBM, no bounce through the CPU), fastsafetensors (4.8-7.5× over the default loader) and the Run:ai Model Streamer (concurrent reads that saturate the disk), plus the simplest lever of all: move fewer bytes (FP8 is half of FP16). This post explains the path, the maths, the 10 knobs, and the cruellest trap: “the second time it loaded fast” is not your loader being good, it is the page cache lying to you. On the generic 4×H100 SXM cluster.

Where you are: the basement, before opening

The analogy: the pantry lift

Let us stay in the restaurant of the series. The shared table was the NVSwitch, the floor next door was NUMA, the maître was the kubelet. All of that describes the restaurant running, with diners seated. But there is a moment no post looked at: before opening, somebody has to bring the entire pantry up from the basement store to the kitchen.

The model weights are the ingredients. They live in the basement store (the disk). The kitchen, the hot line where tokens are plated, is the GPU’s HBM. And between one and the other there is a service lift: the path disk → host → PCIe → HBM. The kitchen cannot serve the first dish until the pantry is upstairs and in place. That restocking time is the cold start.

The intuition trap: the kitchen (HBM) is enormous and blazingly fast, it places ingredients at 3.35 TB/s. So we blame the kitchen when the restaurant opens late. But the kitchen is standing idle waiting for the lift. The bottleneck is never the hot line: it is the lift and the store. And worse still, there is a porter (the default loader) who, instead of carrying whole crates, takes the ingredients out one by one, writes each one down in a notebook and packs them again before sending them up. That porter, not the lift, is half the problem.

The mechanism: what really happens when a model loads

When vLLM starts up with a model in safetensors, the 140 GB of the Llama-70B FP16 make this journey:

Disk → page cache. The kernel reads the .safetensors files from NVMe into the page cache (host RAM). If this is the first time after a reboot, it is a physical read from disk (~14 GB/s on Gen5). If the files are already in page cache from an earlier start-up, this is almost free, and here is where the trap we will see is born.
Deserialise. The default Hugging Face loader mmaps the file and builds the tensors one by one, copying them into a CPU tensor before moving them. This is single-threaded CPU work that saturates neither the disk nor PCIe: most of the “slow” loading time goes here, not on moving bytes.
Host → HBM (H2D). Each tensor is copied from the host buffer into HBM over PCIe Gen5 x16 (~50 GB/s in practice). For the DMA to be efficient, the host buffer should be pinned, which connects directly with the hugepages and pinned memory of the NUMA post.
Place in HBM. HBM receives the 140 GB. At 3.35 TB/s, this takes 42 ms. It is never the bottleneck.

The path has a shortcut: GPUDirect Storage (GDS). Instead of bouncing through the CPU buffer (steps 2-3), a DMA engine close to the NVMe controller writes straight from disk to HBM, without involving the CPU. It is the same principle as the GPUDirect RDMA of the network: get the CPU out of the way. fastsafetensors uses GDS and reaches 26.4 GB/s reading a Llama-70B from NVMe across 4 GPUs.

Why the problem exists: the economics of bytes

The size of the model in bytes is decided by quantisation, and that sets the floor of the cold start, because all those bytes have to move before the first token. For a 70B parameter model:

Format	Bytes/param	Size 70B	Read 1 NVMe @14 GB/s	H2D PCIe @50 GB/s	HBM @3.35 TB/s
FP16 / BF16	2	140 GB	10.0 s	2.8 s	42 ms
FP8	1	70 GB	5.0 s	1.4 s	21 ms
INT4 (GPTQ/AWQ)	0.5	~35 GB	2.5 s	0.7 s	10 ms

Three readings of this table:

HBM never shows up as a problem. The last column is always milliseconds. Anyone saying “the GPU is slow to load” is blaming the wrong place.

Quantising is the most underrated cold start lever. Going from FP16 to FP8 does not only double inference throughput (less HBM bandwidth per token, as we saw in quantization): it also halves the cold start, because there are half as many bytes to push up. It is a two-for-one that sizing usually ignores.

The disk is the bottleneck on bytes; the loader is the bottleneck on time. The figures in the table are the theoretical floor, just moving bytes. The default loader adds single-threaded deserialisation on top, which is the difference between the theoretical 10 s and the real 30-60 s. That is why the solutions attack on two fronts: fewer bytes (quantisation) and a better porter (concurrent loaders / GDS).

The maths that matter: cold start as a tax on autoscaling

Cold start is not paid once. It is paid every time a pod is born. And on an elastic platform, pods are born continuously.

Take an autoscaling setup with KEDA that scales from 2 to 6 replicas when the queue grows. The 4 new replicas take this long to be ready:

$$ T_{\text{ready}} = T_{\text{schedule}} + T_{\text{pull-image}} + T_{\text{load-weights}} + T_{\text{cuda-graphs}} $$

With a Llama-70B FP16 and the default loader, $T_{\text{load-weights}}$ dominates: it can be 40 s of the ~60 s total. During those 40 s, the queue that triggered the autoscaling keeps growing, since the new replicas absorb no traffic until they have loaded. The real number in the formula is not “how many replicas”, it is how long each one takes to start serving, and that number is written by the loading path.

This has a harsh operational consequence: scale-to-zero is unworkable for workloads with a latency SLO if the cold start is 40 s. Nobody waits 40 s for the first token. The real elasticity of an inference platform is not limited by GPU availability, it is limited by how long that GPU takes to have the model inside it. Bringing the cold start down from 40 s to 8 s (with streamer + FP8) is what turns “theoretical scale-to-zero” into “usable scale-to-zero”.

The published numbers give the order of magnitude of the improvement: fastsafetensors cuts start-up from 12.39 s to 4.74 s on a Llama-2-13B over 4×L40S, and from 16.04 s to 6.88 s on 1×A100, 4.8-7.5× over the default deserialiser. The Run:ai Model Streamer loads in 4.88 s from S3 at concurrency 32 and 7.53 s from SSD IO2 at concurrency 8; integrated into vLLM, total time to ready drops to ~23 s from S3. There is no magic here: it is getting the CPU out of the loop (GDS) and reading in parallel (concurrency) to saturate the disk instead of leaving it half idle while one thread deserialises.

The 10 knobs worth touching

Knob 1 — Measure where the time goes (read vs deserialise vs H2D)

Before touching anything: time it. Is the time going on reading from disk, on deserialising, or on the H2D? iostat -x 1 during the load tells you whether the NVMe is saturated (disk bottleneck) or nearly idle (loader/CPU bottleneck). If the disk is at 20%, the problem is not the disk: it is the porter. Changing disk would fix nothing; changing loader would.

Knob 2 — `--load-format`: choosing the porter

vLLM exposes several loaders via --load-format: safetensors (default), runai_streamer, fastsafetensors, tensorizer. The default is the slowest. Changing one flag can be the cheapest 4-7× there is.

# Run:ai Model Streamer (concurrent reads, saturates the disk)
vllm serve meta-llama/Llama-3.1-70B --load-format runai_streamer
# fastsafetensors (GPUDirect Storage, direct DMA disk→HBM)
vllm serve meta-llama/Llama-3.1-70B --load-format fastsafetensors

Knob 3 — Streamer concurrency

The Run:ai Model Streamer splits the read across N threads according to the size of each tensor in order to saturate the storage bandwidth. Concurrency is the key parameter: 16 is usually enough for local NVMe; 32 (sometimes 64) for high-throughput network storage. One thread does not saturate a Gen5 NVMe; 32 do.

vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

Knob 4 — GPUDirect Storage (GDS) + fastsafetensors

If there is an nvidia-fs driver, a supported filesystem and local NVMe, GDS writes straight disk→HBM with no bounce through the CPU. It is the difference between the two paths in the diagram. But: it requires the stack to be in place, and it only wins if the bottleneck is the CPU bounce, not the disk itself. Check with gdscheck.

Knob 5 — Local NVMe for the weights, not network

Serving the weights from Ceph RGW / NFS is convenient (one shared place) but it puts the network into the loading path. For cold start, weights on the node’s local NVMe (or a local cache). Network storage is for the model repository; the inference node should have a hot local copy.

Knob 6 — Pre-pull / local model cache

An initContainer that downloads the model onto a local or hostPath NVMe volume before starting vLLM turns a cold start “from the network” into one “from local NVMe”. Combined with a per-node cache DaemonSet, new pods on an already warm node read from local disk, not from the network.

Knob 7 — Quantisation: move fewer bytes

Weights already in FP8 or INT4 on disk = half or a quarter of the cold start. It is the knob from the table above. And it composes with all the others: FP8 + streamer + GDS is multiplicative.

Knob 8 — Parallel loading across GPUs (vLLM V1)

vLLM’s V1 engine (the default since 0.19) loads the weight shards in parallel across the GPUs of a TP group, instead of sequentially. At TP=4, each GPU loads its quarter at the same time. Check it is active; in old versions loading was serial and the cold start of TP=4 was almost 4× that of TP=1.

Knob 9 — NUMA locality of the NVMe

The NVMe hangs off a PCIe root under a specific socket, exactly the same NUMA map as the host post. If the load’s host buffer lands on the wrong socket, the H2D crosses the UPI. The fifth list to align, alongside isolcpus, reserved-cpus and the NIC IRQs: which socket is local to the NVMe and to the destination GPU. nvidia-smi topo -m shows it.

Knob 10 — Do not pay the cold start: keep pods warm

Sometimes the answer is not to load faster, but not to unload. A floor of always-live replicas (no scale-to-zero), or a pool of preloaded warm standbys, swaps “wait 40 s” for “0 s”. It is the cost of idle GPU in exchange for start-up latency: a capacity planning decision, not a technical one.

Summary table

#	Knob	What it attacks	Risk / cost
1	`iostat` while loading	knowing whether the bottleneck is disk or loader	none; always do it first
2	`--load-format`	single-threaded deserialisation	format compatibility
3	streamer concurrency	underused disk	host RAM for buffers
4	GPUDirect Storage	bounce through the CPU	requires nvidia-fs + supported FS
5	local NVMe vs network	the network in the path	duplicating weights per node
6	pre-pull / node cache	network on every start-up	local disk space
7	FP8/INT4 quantisation	bytes to move	quality (measure, do not assume)
8	V1 parallel loading	serial loading across GPUs	none if V1 is active
9	NUMA of the NVMe	H2D crossing the UPI	align with the rest of the NUMA lists
10	warm pods / no zero	the whole cold start	idle GPU paid for

How it connects with the rest of the stack

With autoscaling. The whole of scale-from-zero with KEDA rests on this: real elasticity is limited by the cold start, not by GPU availability. An autoscaler with 40 s of loading reacts late to every peak.

With disaggregated serving. In disaggregated prefill/decode, bringing up a decode pool on demand pays the cold start of loading the model into each new pod. The elasticity of the pattern depends on how fast those pods start.

With canary/blue-green. Every canary deployment loads a new version of the model alongside the old one. The validation time of a canary includes its cold start; large models make deployments slower and more expensive.

With NUMA and hugepages. The load’s host buffer wants to be pinned and NUMA-local, the same thing the host post asked for on the hot path. The loading path is another client of the same NUMA map.

With quantisation. FP8/INT4 is not only inference throughput: it is the direct lever on the bytes of the cold start.

With capacity planning. Sizing that ignores cold start underestimates how many replicas are needed to absorb a peak: if they take 40 s to start, you need a bigger permanent cushion.

Traps and things that are not what they seem

“The second time it loaded really fast.” This is the star trap. The first load filled the page cache (host RAM); the second reads from RAM, not from disk, and flies. But in production pods are ephemeral and are born on different nodes: the start-up that counts is the cold one, on a node where those files are not in page cache. Benchmarking the second load is measuring a situation that almost never happens at the moment that matters (the peak that triggers the autoscaler).

“GDS always speeds things up.” No. GDS removes the bounce through the CPU; if your bottleneck is the disk itself (saturated NVMe) or the deserialisation, GDS does not touch that part. Measure first (knob 1). It also demands nvidia-fs, a supported filesystem, and sometimes it does not work over whatever network storage you have.

“mmap makes loading instant.” mmap maps the file but reads nothing yet: the cost is deferred to the first access to each page. The time does not disappear, it moves, and the first token pays the page faults that start-up did not. You have moved the cold start into the latency of the first request, which is probably a worse place to have it.

Weights on network storage “because it is cleaner”. Sharing a model repository on Ceph RGW is fine for storing them; serving the cold start from there puts the network (and its latency and its contention) into the critical path. Local NVMe cache on the inference node.

Loading FP16 and quantising at start-up. Quantising on the fly during loading (for example FP16→FP8 on the GPU) can be slower than having the weights already quantised on disk: you move twice the bytes and on top of that you do conversion work. If you are going to serve in FP8, store the weights in FP8.

Optimising the load and ignoring T_cuda-graphs. Bringing weight loading down to 8 s and forgetting that CUDA graph capture adds several more seconds leaves the cold start half done. That second half of start-up is the subject of the next post.

Conclusion

The whole series optimised what happens with every token: the cable, the host, the network, the silicon. But before the first token there is a journey almost nobody times and that decides whether an inference pod opens in 10 s or in 60 s: pushing the model from disk to HBM. Intuition blames the GPU, and the GPU is innocent: HBM swallows 140 GB in 42 ms. The bottleneck is the supply chain: a disk reading at 14 GB/s, a PCIe copying at 50, and above all a default loader that deserialises tensor by tensor with a single thread and turns 10 s of bytes into 60 s of waiting. The solutions attack the two right fronts, fewer bytes (quantisation) and better transport (GDS, concurrent streamers, local NVMe), and give 4-7× almost for free. And above the technique, an idea that reorders the priorities: on an elastic platform, cold start is not a start-up detail, it is the ceiling on elasticity. The fastest GPU in the world does not scale if it takes 40 s to have the model inside. The pantry lift, the one nobody timed, is what decides what time the kitchen really opens.

References

NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/.
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/.
vLLM, Loading model weights with fastsafetensors: https://docs.vllm.ai/en/stable/models/extensions/fastsafetensor/.
foundation-model-stack, fastsafetensors (high-performance loader, GDS): https://github.com/foundation-model-stack/fastsafetensors.
Speeding up Model Loading with fastsafetensors (arXiv 2505.23072): https://arxiv.org/html/2505.23072v1.
NVIDIA, GPUDirect Storage: A Direct Path Between Storage and GPU Memory: https://developer.nvidia.com/blog/gpudirect-storage/.
NVIDIA, Magnum IO GPUDirect Storage (benchmarking and configuration): https://developer.nvidia.com/gpudirect-storage.