Storage in the AI era (1/4): the state of the art

Contents

Public conversation about AI infrastructure almost always revolves around the GPU: how many FLOPs, how much HBM, how many watts. But anyone who designs or runs an AI factory knows the bottleneck has moved. A Blackwell GPU sitting idle waiting for data costs exactly the same as a Blackwell GPU doing work; what makes the difference is the subsystem feeding it. This first article in the series maps the state of the art in AI storage as of mid-2026: the memory and storage hierarchy, the media technologies (NAND, NVMe, CXL), the parallel file systems that dominate GPU clusters and the new inference stack that has turned the SSD into a component of the memory hierarchy.

The three articles that follow go deeper into the three properties that matter to an architect once that storage reaches production: performance, security and availability.

The underlying problem: the memory wall

A modern GPU is, above all, a machine for moving data. Compute itself is almost never the limit; the limit is the bandwidth feeding the functional units. The hierarchy that runs from GPU registers down to NVMe spans six orders of magnitude in latency and several in bandwidth. Registers are on the order of a million times faster than an NVMe access, and in between sit on-chip SRAM, HBM, host DRAM and, finally, persistent storage.

The practical consequence is well known: in many training workloads GPUs run at 30-50 % of their theoretical FLOPs utilisation because they spend a significant fraction of their time waiting for data. This is the memory wall. Its most useful quantitative formulation is arithmetic intensity, the ratio of operations to bytes moved:

$$I = \frac{\text{FLOPs}}{\text{bytes}}$$

When a workload’s intensity falls below the hardware’s break-even point (the ridge point of the roofline model), performance is limited by bandwidth rather than compute. The industry’s race towards HBM, towards NVLink and, at the layer that concerns us, towards 28 GB/s NVMe and 800 Gb/s networks, is at bottom a war against that wall.

The problem gets worse because compute has grown much faster than memory bandwidth. Each GPU generation multiplies FLOPs more generously than the TB/s of memory that feed them, so the ridge point shifts towards ever higher arithmetic intensities: more workloads fall on the bandwidth-limited side. This is the structural reason storage and memory have moved to the foreground. It is not that storage has improved less (it has improved enormously), but that the GPU demands more of it every year. An architect who designs with FLOPs alone in mind will build systems that spend half their time waiting.

It pays to keep the whole pyramid in view when we talk about “storage” in AI, because design decisions at one level constrain the others:

LevelTechnologyTypical bandwidthLatencyCapacity per device
On-packageHBM3e / HBM41.2-3.3 TB/s per stackns36-64 GB
Host memoryDDR5 / CXL0.3-0.5 TB/s~100 nsTB
Hot flashNVMe Gen5/Gen614-28 GB/s10-100 µs4-256 TB
CapacityNearline QLC / HDD1-7 GB/sms30-245 TB
ArchiveObject / tapevariablesEB (aggregate)

The state of the art in 2026 consists, to a large extent, in how the boundaries between these levels blur: HBM spills over into DRAM, DRAM into CXL, and the inference KV-cache into NVMe. Storage has stopped being the basement of the hierarchy and has become part of it.

HBM4: the ceiling rises, but it stays expensive

At the top sits high-bandwidth memory stacked next to the GPU. In April 2025 JEDEC published the HBM4 standard (JESD270-4), which doubles the interface to 2048 bits per stack against HBM3e’s 1024 bits, reaches up to 8 Gb/s per pin and lands around 2 TB/s typical with peaks up to 3.3 TB/s per stack. It supports configurations of 4 to 16 layers with 24 or 32 Gb dies, up to 64 GB per cube. HBM3e, the generation that dominates accelerators in production in mid-2026, sits near 1.2 TB/s and a 36 GB maximum.

The figure that matters to an architect is not peak bandwidth but scarcity. Micron has essentially sold out its entire HBM production through 2026, and capacity expansion is not expected before late 2027-2028. That constraint is what pushes the load downwards through the hierarchy: if you cannot add HBM, you have to learn to spill gracefully into DRAM, CXL and flash. Much of the recent innovation comes from there.

NAND and SSD: capacity explodes, the interface widens

The flash layer is where AI’s pressure on the market shows most. Two vectors at once: capacity and interface.

On capacity, 2025-2026 has been the year of the race to 245 TB. The Solidigm D5-P5336 reached 122 TB (PCIe 4.0) and became the highest-capacity SSD available, on sale for around 12,400 USD; Solidigm has confirmed models of 245 TB or more for late 2026. Micron introduced the 6600 ION with 245 TB in E3.L form factor. By mid-2026 there are at least eight 245 TB SSDs announced by SanDisk, Samsung, Kioxia, Micron, SK hynix and others, all built on high-density QLC NAND (SK hynix is already producing 321-layer QLC) to satisfy hyperscaler demand. Enterprise QLC is consolidating as a replacement for nearline HDD, whose supply lags well behind demand and whose lead times have stretched.

QLC deserves a qualification that sometimes gets lost in the excitement over capacity: write endurance. QLC NAND stores four bits per cell, which multiplies density but reduces the number of erase cycles it tolerates compared with TLC. For AI datasets written once and read many times, the typical case of a training set, this is irrelevant and QLC is the right choice. For workloads with intensive, constant writes, certain checkpoints or a fast-rotating KV-cache, wear has to be watched and TLC often reserved. Knowing the write pattern of each workload is what allows the right medium to be assigned without overpaying or under-wearing.

On interface, PCIe Gen6 has entered datacenter production. The Micron 9650 was the first mass-produced PCIe Gen6 datacenter SSD (February 2026), with 28 GB/s of throughput, roughly double Gen5. Controllers are keeping pace: the Silicon Motion SM8466 advertises 28 GB/s sequential and 7 million random IOPS; FADU’s Gen6 part (“Lhotse”) targets 28.5 GB/s read, 6.9 million IOPS, NVMe 2.2 and native SR-IOV with 128 virtual functions. The adoption path is clear: Gen6 arrives first in AI, HPC and the datacenter, and only afterwards, if at all, in the consumer PC.

Over the network, NVMe-oF splits into two worlds. NVMe/TCP leads greenfield array-to-host deployments for its operational simplicity, with latencies of 100-200 µs. NVMe over RDMA (RoCEv2, iWARP) remains the gold standard for low latency (10-20 µs) where a lossless fabric already exists, such as the ones AI clusters deploy. Ethernet at 25-100 Gb/s is steadily eroding Fibre Channel’s share of storage traffic.

The choice between TCP and RDMA is not an aesthetic preference. NVMe/TCP runs over any existing Ethernet network and greatly simplifies operations, at the cost of higher latency and of host CPU consumption that, under heavy load, competes with compute. NVMe/RoCE demands a lossless fabric, with priority flow control and congestion control properly tuned, plus capable NICs, but it delivers the latency that latency-sensitive AI workloads need and offloads processing to the NIC. In an AI cluster that already deploys InfiniBand or RoCE for GPU-to-GPU traffic, reusing that fabric for storage is the natural decision; in a mixed enterprise infrastructure, NVMe/TCP usually wins on operability.

CXL: from PowerPoint to production

Compute Express Link spent years as the eternal promise. By early 2026 it has finally reached mainstream adoption in production platforms with CXL 3.0/3.1, aligned with PCIe 6.1. The hardware support is there: Intel Xeon 6 (Granite Rapids and later) includes native CXL 3.0 in the CPU, and AMD EPYC Turin is a target platform. CXL 3.0 multi-host pooling switches are already running at some colocation providers.

The use case that has moved from theory to numbers is memory tiering: keeping hot data in GPU HBM and pushing cold data to a shared CXL DRAM pool. Supply chain vendors (Astera Labs, MemVerge, XConn) report notable gains from offloading the inference KV-cache to CXL memory compared with SSD or RDMA-based solutions. The specific figures deserve caution, since they come largely from the manufacturers themselves, but the direction is real: CXL has become a usable disaggregated memory layer, not a laboratory experiment. Tiering support in the Linux kernel (6.1 onwards) is maturing, although getting value from it still requires NUMA-aware applications.

Parallel file systems: the GPU battlefield

If the media layer is where silicon vendors compete, the parallel file system layer is where platform vendors compete, and it is the one an AI architect touches daily. The market has reshuffled in a few years.

WEKA has probably been the most talked-about case. Its system (WekaFS, now repackaged as NeuralMesh) bypasses the Linux kernel by running as an RTOS in user space, uses RDMA instead of TCP/IP and distributes metadata through consistent hashing. Published figures cite latencies of 100-200 µs against more than 1000 µs for traditional stacks, and metadata performance that doubles Lustre’s. In 2026 it launched its NeuralMesh AI Data Platform on the NVIDIA reference design, with an “Augmented Memory Grid” aimed at context memory for agentic AI.

VAST Data has bet on a different architecture, DASE (Disaggregated Shared-Everything), a single all-flash tier on which it has built what it calls an “AI OS”: a stack with a database, a data engine, InsightEngine for real-time AI pipelines, vector search and RAG, and integrated governance. Azure has adopted its system, a sign of the traction it has gained.

Lustre (DDN) remains the most widespread file system in classic supercomputing, but it is being eroded in the GPU arena by VAST and WEKA. DAOS shines at the top end of traditional HPC but barely appears in commercial AI clusters. IBM Storage Scale (the former Spectrum Scale/GPFS) has repositioned itself with Content Awareness for RAG and support for NVIDIA’s data platform, alongside a refresh of the FlashSystem range.

The common denominator of this whole generation is alignment with NVIDIA. Platforms are certified against DGX SuperPOD and integrate with GPUDirect Storage, which allows direct DMA between storage and GPU memory, skipping the CPU and the bounce buffer. In GB200 NVL72 architectures, a liquid-cooled rack that presents 72 GPUs as a single giant GPU through NVLink, with 13.4 TB of unified GPU memory, GPUDirect Storage certification stops being an ornament and becomes a requirement.

To place the main players side by side, it helps to summarise their architectural bets, which are genuinely different and not mere variations on a single design:

PlatformArchitectureDifferentiatorFit in AI
WEKA NeuralMeshUser-space RTOS, RDMAVery low latency, metadata distributed by hashingTraining and agentic inference, context memory
VAST DataStoreDisaggregated all-flash DASESingle exabyte-scale multiprotocol namespace, AI OSLakehouse + RAG + inference
IBM Storage ScaleParallel (ex-GPFS)Content Awareness for RAG, HPC maturityEnterprise and HPC with unstructured data
DDN EXAScaler/LustreHardened LustreRaw throughput per rack, massive deploymentsSupercomputing and AI factories
Pure StorageAll-flash with PurityEvergreen operations, simplicityEnterprises that value operations over peak

The choice between them is rarely decided by peak GB/s, but by workload pattern (sequential training against random inference), by the operating model and by the degree of integration required with the NVIDIA stack. A sensible architect first assesses their workload mix and only then compares numbers.

The rack as the unit of compute: GB200, Rubin and BlueField-4

The change in granularity is worth pausing on. For decades the design unit was the server; today, in AI, it is the rack. The GB200 NVL72 integrates 72 Blackwell GPUs and 36 Grace CPUs into a single 130 TB/s NVLink domain, presented to software as one logical GPU with 13.4 TB of unified memory and capable of up to 30 times the real-time inference of a trillion-parameter LLM against the previous generation. At that scale storage does not compete with internal bandwidth (it could not), but it must feed model loading and the inference KV-cache (and, where applicable, training checkpointing) without strangling the 72 GPUs that behave as one.

The roadmap sharpens the trend. The Rubin generation and, in particular, the Rubin CPX, a GPU specialised in long-context inference with more than a million tokens, push still more data towards the memory and storage layers. And the piece that stitches it all together is the DPU: NVIDIA’s BlueField-4 enables an “Inference Context Memory Storage” platform that turns the KV-cache into a pod-level shared flash resource, connected over Spectrum-X Ethernet. The DPU offloads storage and security data movement from the CPU, and is becoming the true storage controller of the AI node. For the architect, this means storage design no longer ends at the array: it includes the NIC, the DPU and the fabric.

The inference stack: when the SSD joins the memory hierarchy

The most important conceptual change of the past two years, and the one that best summarises the state of the art, is this: flash storage has become part of the inference memory hierarchy. The cause is context windows. As models handle hundreds of thousands, or millions, of tokens, the KV-cache generated by the prefill phase overflows HBM. That overflowing KV-cache drops first into host DRAM and then to the NVMe SSD.

NVIDIA has standardised this pattern. Its Dynamo orchestration layer operates across the whole hierarchy, HBM → CPU DRAM → NVMe → network storage, with a KV-cache offload engine and a transfer library (NIXL) for sharing cache between nodes. In January 2026 it standardised KV-cache offload to NVMe SSD, and it has introduced an “Inference Context Memory Storage” platform based on BlueField-4 DPUs that turns the KV-cache into a high-bandwidth shared resource at pod level. Solidigm and other SSD makers have aligned their message around the same idea: long-context inference is, in part, a flash storage problem.

The design consequences are concrete. A 2026 inference server carries between 8 and 32 SSDs, with the figure expected to reach 32 within a couple of years, and a random access pattern, against the sequential pattern and the dozen or so drives of a training server. The ratio of inference to training servers can reach 50 to 1. The centre of gravity of storage investment is shifting towards inference.

Anatomy of storage in an inference factory

It is worth pausing here, because an inference AI factory, a system designed to serve models in production rather than train them, has a storage profile of its own, different from the training cluster that dominates the literature. While training is a batch job that tolerates stoppages and is measured in weeks, inference is a continuous service, latency-sensitive and bound by an SLA: every millisecond and every failure shows up at the client. Its storage is organised into four planes that are best designed separately.

The first plane is the model registry. The weights of the models being served, which may number in the dozens, with versions, quantised variants and LoRA adapters, live in object storage or a shared file system, and are loaded into GPU HBM when a model is activated. The critical requirement here is not sustained throughput but load time: when a new replica has to start or a GPU has to switch models, reading tens or hundreds of GB of weights determines the cold start. A slow registry translates into sluggish autoscaling and GPUs that take their time entering service, exactly when demand rises.

The second plane is the KV-cache hierarchy, the heart of modern inference performance. As we saw, the KV-cache overflows HBM and drops to DRAM, CXL and NVMe. In an inference factory this is not a marginal detail: it is a first-class storage layer, with its own sizing, its random access pattern and critical latency, and a lifetime measured in seconds. NVIDIA’s standardisation of KV-cache offload to NVMe SSD and the ICMS platform on BlueField-4 exist precisely for this: to allow long contexts and cache reuse across requests (and across nodes) without paying the price of recomputation or of adding HBM, which is scarce and expensive.

The third plane is the knowledge store for RAG: the vector databases (Milvus, Weaviate, Pinecone, or S3 Vectors) and the source documents the system retrieves to enrich its answers. The pattern here is read-intensive, with similarity searches over indexes that can run to TB and are best kept on low-latency flash. The availability and freshness of this store determine answer quality as much as the model itself, and updating it, reindexing and taking in new documents, is a constant write flow the design has to account for.

The fourth plane is observability and logging: traces, request and response logs, metrics and, where regulation demands it, the auditable archive of interactions. It is a high-volume append-only write flow that usually ends up in object storage, and one that a GPU-centric design tends to forget about until it saturates something.

The conclusion for anyone building an inference factory is that its storage looks nothing like a rescaled training cluster. It is sized by model load time, by KV-cache capacity and latency, by vector search performance and by telemetry volume, with a cross-cutting imperative of continuous service under SLA. Training, where it exists, is the occasional guest; inference is the permanent resident.

The other end: object, lakehouse and vectors

Not everything is hot flash. The training database and the system of record live in object storage. The novelty of 2025-2026 is convergence with open table formats: Apache Iceberg, Delta Lake and Hudi bring ACID transactions, time travel and schema evolution on top of cheap object storage. AWS has launched S3 Tables (managed Iceberg, with faster queries) and S3 Vectors (native vector storage with savings of up to 90 % for AI workloads). Vector databases, Pinecone, Milvus, Weaviate, integrate with data lakes to feed RAG, and columnar formats such as Parquet, together with Lance for multimodal data, are consolidating as the substrate of the lakehouse.

Computation near the data (computational storage) remains active R&D rather than mass deployment. The corresponding NVMe specifications have been published since late 2023, with eBPF as the execution environment for downloadable functions, but real adoption is still marginal. The idea, running filtering, decompression or search directly on the device so the data never travels to the CPU, is seductive for AI workloads that read enormous datasets at low arithmetic intensity, but the programming model and the lack of tooling standardisation hold deployment back. It is a technology to watch, not yet one to put into production.

The data lifecycle: tiering as an architectural decision

Taking in the whole picture, it is clear that no data lives in a single place for its entire lifecycle. A dataset arrives raw in the object data lake, is transformed and catalogued in an open table format (Iceberg, Delta), is promoted to the hot flash layer of the parallel file system to feed training, generates checkpoints that live for days on NVMe, and finally, together with the resulting models, is archived to cold object storage or tape. Automatic tiering between these layers has gone from a cost optimisation to a first-order architectural decision, especially amid NAND and HBM scarcity.

The practical rule that emerges is to place every byte in the cheapest tier that satisfies its latency and bandwidth requirement at that moment, and to automate the transitions. Hot flash (NVMe Gen5/Gen6) absorbs the training working set and the inference KV-cache; capacity QLC replaces nearline HDD for datasets that are read sequentially; object storage is the durable, versioned system of record; and tape is resurging as an active archive at exabyte scale to fight the cost of flash. Designing those boundaries, and measuring how much data crosses each one, is now part of the AI storage architect’s job, not the administrator’s.

The market context: a supply supercrisis

None of this happens in an economic vacuum. Storage demand from AI servers is 8 to 10 times that of a traditional server, and it is growing above 20 % a year, while global NAND supply grows only 15-17 %. The result, in the words of several analysts, is a memory “supercrisis”: high double-digit contract price rises for DRAM and NAND in 2026, stretched lead times and collateral effects on smartphones and PCs. For an architect, this turns capacity efficiency (QLC, compression, erasure coding, tiering) into a first-order cost lever rather than an implementation detail.

What this changes for the architect

If the state of the art had to be distilled into a handful of practical implications, these would be them. First, storage is designed by workload pattern, not by capacity: a training server and an inference server have opposite requirements, sequential against random, few drives against many, and mixing them in a single template is a common mistake. Second, the limit almost never sits in the medium any more but in the full chain of medium, NVMe, DPU, NIC, fabric and file system; optimising one link while leaving the rest untouched does not move the needle. Third, supply scarcity turns capacity efficiency (QLC, compression, erasure coding, tiering) into a cost lever comparable to buying the drives themselves. And fourth, the boundary between memory and storage has dissolved: thinking in terms of “RAM on one side, disk on the other” no longer describes a modern AI system, where HBM, DRAM, CXL and NVMe form a software-managed continuum.

Takeaways

The state of the art in AI storage in 2026 comes down to one idea: the hierarchy has become continuous. HBM spills into DRAM, DRAM is disaggregated over CXL, the inference KV-cache drops all the way to NVMe, and the parallel file system orchestrates the whole so the GPU never waits. Media technologies (HBM4, 245 TB QLC, NVMe Gen6) raise the ceilings, but supply scarcity turns efficiency into a design constraint. And the software battlefield, WEKA, VAST, IBM, DDN, is fought over a single metric: keeping the GPUs fed.

With this map on the table, the three articles that follow go into the detail of the properties that decide whether a storage architecture works in production.

See also

Sources