Storage in the AI era (2/4): performance

Contents

In the first article we drew the map: the memory-storage hierarchy, the media technologies and the software stack that dominates GPU clusters. This second article goes into the property that justifies all that complexity: performance. The question an architect has to answer is not “how many GB/s does my array deliver?”, but “can I get my GPUs working at 95 % instead of 40 %?”. That difference is, literally, half the compute bill.

The metric that really matters: accelerator utilisation

It is tempting to measure AI storage with the classic metrics, sequential throughput, random IOPS, latency, and we do need them. But none of them captures what the business pays for. The integrating metric is accelerator utilisation: what fraction of the time the GPU spends computing rather than waiting for data.

Most real ML pipelines run at between 30 % and 60 % GPU utilisation. Industry surveys are damning: only around 7 % of AI/ML teams manage to exceed 85 % at peak, and poor forecasting or weak autoscaling can leave GPUs idle 70-85 % of the time. A job running at 30 % utilisation is paying for 70 % of its compute capacity and getting nothing for it.

We can formalise this. If \( t_{\text{cmp}} \) is the compute time per batch and \( t_{\text{io}} \) the time the GPU spends blocked waiting for I/O that fails to overlap with compute, effective utilisation is:

$$U = \frac{t_{\text{cmp}}}{t_{\text{cmp}} + t_{\text{io}}}$$

The goal of all AI storage engineering is to drive \( t_{\text{io}} \) towards zero, whether by increasing bandwidth, reducing latency or overlapping I/O with compute through prefetching and pipelining. And there is an aggravating factor: the faster the GPU, the harder it is to feed. Moving from V100 to H100 cut the per-batch compute time of the 3D U-Net model by 76 %, turning a bandwidth-sensitive workload into a latency-sensitive one. Every accelerator generation raises the bar for storage.

This change of regime, from bandwidth-limited to latency-limited, has a design consequence worth internalising. When a workload is bandwidth-limited, the fix is to add transfer capacity: more NVMe, more lanes, more nodes. When it becomes latency-limited, that recipe stops working; what matters then is response time per operation, which is attacked with cache, with intelligent prefetching and with data paths that remove hops (GPUDirect, for instance). Diagnosing correctly which of the two regimes each workload sits in is the first task of any optimisation: throwing bandwidth at a latency-limited workload is spending money without moving utilisation.

MLPerf Storage: the benchmark that does measure the right thing

For years there was no neutral way to compare AI storage platforms. MLPerf Storage, from MLCommons, has provided one. Its design insight is not to measure GB/s in the abstract but to simulate the accelerators’ think time so as to generate a realistic I/O pattern, and to require those simulated accelerators to sustain a minimum utilisation level. In other words, it measures exactly what matters: how many accelerators a platform can keep fed without their utilisation dropping below the threshold.

v1.0 (September 2024) used the 3D U-Net, ResNet-50 and CosmoFlow models, simulating A100 and H100, with utilisation thresholds of 90 % for the first two and 70 % for CosmoFlow. v2.0 (August 2025) marked a leap: more than 200 results from 26 organisations, including Alluxio, DDN, Hammerspace, HPE, IBM, KIOXIA, Micron, Oracle, Samsung and WDC, systems able to sustain roughly twice as many accelerators as in v1.0, and, above all, a new checkpointing workload (which we will come back to).

Some v2.0 results help calibrate the state of the art:

SystemThroughputAcceleratorsGPU utilisation
Volumez1.079 TB/s—92.21 %
Hammerspace (5 nodes)420.8 GB/s140>94.7 %
Hammerspace (3 nodes)253.1 GB/s84>94.7 %
Hammerspace (1 node)85.6 GB/s28>94.7 %
Alluxio24.14 GiB/s12899.57 %

The Hammerspace figure illustrates a property an architect should demand: linear scaling. From 1 to 5 nodes throughput grows from 85.6 to 420.8 GB/s while keeping utilisation above 94.7 % with a minute coefficient of variation (0.08-0.14 %). Linear scaling is what separates an AI platform from a fast NAS.

The numbers from commercial platforms

Outside the benchmark, platforms certified for DGX SuperPOD publish aggregate figures worth keeping as an order-of-magnitude reference:

  • WEKA WEKApod Nitro: 720 GB/s read and 186 GB/s write per configuration, 18 million IOPS, with ConnectX-8 NICs at 800 Gb/s. Per node, 70 GB/s read and 40 GB/s write; the minimum 8-node configuration delivers 560 GB/s read. A single entry-level system covers the I/O demand of a 1,152-GPU GB200 scalable unit.
  • DDN AI400X2: more than 90 GB/s and 3 million IOPS per appliance; the Turbo variant saturates close to 100 GB/s (800 Gbps) per DGX B200. DDN’s new platform claims more than 1 TB/s read per appliance with per-rack scaling, and has fed NVIDIA’s Eos supercomputer at 4 TB/s.
  • VAST Data: multiple TB/s from a single mount point via NFS multipathing, RDMA over NFS and GPUDirect, managing exabytes in a single multiprotocol namespace.
  • Pure Storage: certified for DGX SuperPOD with GB200 and GB300.

These figures only mean something in relation to the cluster they feed. A GB200 NVL72 moves 130 TB/s of internal GPU-to-GPU communication; storage does not compete with that, but it does have to sustain data loading and checkpointing without strangling 72 GPUs that behave as one.

The I/O pattern of training and GPUDirect Storage

Understanding the I/O pattern is the key to neither over- nor under-sizing. Training combines several patterns: large sequential reads of the dataset, small random reads during shuffling between epochs, and massive periodic writes during checkpointing. Each stresses storage differently.

The traditional data path goes through the CPU: from storage to a bounce buffer in host memory and from there to GPU memory. Every byte of every checkpoint and every weight load pays the latency and CPU overhead of that double hop. GPUDirect Storage (GDS) cuts it out: it enables direct DMA between GPU memory and NVMe through the cuFile user-space library and the nvidia-fs kernel module, which intercepts POSIX calls and redirects them through the DMA engine. The result is zero-copy, with direct transfers exceeding 40 GB/s and, above all, without stealing cycles from the CPU.

There is a nuance that naive deployments miss: GDS is optimised for large, aligned, sequential transfers. For workloads dominated by small files or heavily random access, the traditional path with CPU caching can be more efficient. GDS is not a magic “more performance” switch; it is a tool for a specific pattern. Designing the data pipeline, meaning dataset format, shard size and number of DataLoader workers, matters as much as picking the array.

The data pipeline deserves a pause, because it is where performance leaks most quietly. Training does not read the dataset just any way: it walks it in batches, shuffles it between epochs to avoid ordering bias, and often applies transformations (image decoding, tokenisation, augmentation) on the CPU before handing the tensor to the GPU. Each of those steps can become the bottleneck. Too few DataLoader workers leaves the GPU waiting; a dataset format that forces millions of small files open sinks metadata performance; a badly designed shuffle turns efficient sequential reads into costly random accesses. Common practice in serious workloads is to pack the dataset into large shards (formats such as WebDataset, TFRecord or Parquet) precisely so that the read pattern is large and sequential, the terrain where GDS and parallel file systems perform best. The mental rule: the fastest storage in the world will not save a badly designed data pipeline.

The real cost of checkpointing (when there is training)

Checkpointing is the most demanding write workload in the AI world, but it needs to be placed properly: it is a training problem. A pure inference factory barely suffers from it, since its heavy writes are different ones, KV-cache, vector indexes, telemetry. We cover it here because many installations combine some fine-tuning or occasional training with serving, and because it illustrates better than any other case how asynchrony beats brute force. If your platform is for inference, you can read this section as context and jump to the next one, which is the one that really concerns you.

That said, most people’s intuition about checkpointing is wrong on two counts: size and frequency.

Size. A checkpoint is not just the weights. The rule of thumb is around 16 bytes per parameter, because the bulk is optimiser state. The figures from the MLPerf Storage v2.0 checkpointing workload, based on Meta’s Llama 3 Herd, make it plain:

ModelProcessesCheckpoint sizeOf which, optimiser state
8B8105 GB90 GB
70B64912 GB—
405B5125.29 TB—
1T102415 TB13.2 TB

Frequency. At scale, hardware failures are constant (we develop this in the article on availability). Meta’s model for a 16,000-accelerator cluster predicts a failure every few hours. To lose less than 5 % of progress between failures, you need on the order of 20 checkpoints in that interval. That translates into aggressive cadences:

  • 16,000 accelerators: around 155 checkpoints a day, one every 9.3 minutes.
  • 100,000 accelerators: around 967 checkpoints a day, one every 1.5 minutes.

And here is where the bandwidth problem appears. A synchronous checkpoint halts training while it writes. If \( S_{\text{ckpt}} \) is the checkpoint size and \( t_{\text{win}} \) the acceptable time window for writing it while keeping overhead below 5 %, the required bandwidth is:

$$B_{\text{req}} = \frac{S_{\text{ckpt}}}{t_{\text{win}}}$$

For a 1T model on 100,000 accelerators, the window falls to around 4.4 seconds. Writing 15 TB in that time demands on the order of 3.6 TB/s across the cluster, or some 200 GB/s sustained over the whole job. On checkpoints alone, that scenario generates more than 14 PB of writes a day. This is not a marginal workload: it is one of the most demanding write workloads in existence.

It is worth seeing the calculation laid out, because it reveals where the lever is. Take the 405B model, whose checkpoint weighs 5.29 TB. If policy requires writing it while keeping overhead below 5 % at a cadence where the acceptable window is, say, 30 seconds, the synchronous bandwidth needed is around 176 GB/s sustained, within reach of a high-end array, but only one dedicated to that task. If instead of 30 seconds the window contracts to 5 (a larger cluster, more frequent failures), the requirement jumps above 1 TB/s. Sensitivity to the window is brutal: the difference between a viable design and an impossible one lies in how time is managed, not in how many drives are bought. That is where asynchrony changes the rules.

The solution is not just more bandwidth; it is asynchrony. Asynchronous checkpointing decouples the snapshot (a fast GPU→CPU copy) from persistence (a background write, overlapped with compute). PyTorch’s Distributed Async Checkpointing cuts effective checkpoint time by 10 to 20 times; in one IBM example a 7B model brought its down time down from 148.8 to 6.3 seconds, a 23.6-fold improvement. Techniques such as CheckFreq pipeline the snapshot and persistence with training, allowing a checkpoint every 14-19 iterations. This is why vendors like VAST argue the relevant metric is not GB/s but checkpoint overlap: what fraction of the checkpoint overlaps with compute. Around 10 % overlap is usually enough.

The design lesson is clear: sizing training storage for the peak write rate of synchronous checkpointing is expensive and often unnecessary if the software stack gets asynchrony right. But trusting asynchrony without measuring overlap is playing roulette with weeks of compute.

Performance in an inference factory

Here is the heart of the matter for anyone running an inference factory. In an inference factory storage is not measured in epochs or GPU-hours but in the experience the client perceives, and that experience condenses into two metrics: TTFT (time to first token, the time until the first token of the answer) and inter-token latency (the rate at which the following tokens arrive). At fleet scale, the metric that pays the bills is throughput in tokens per second per GPU, because it determines how many requests each accelerator serves. Storage touches all three, in ways a training-centric design does not anticipate.

Prefill, decode and their disaggregation

Inference has two phases with opposite profiles. Prefill processes the entire input prompt at once, generates the key-value pairs of the KV-cache and is a compute-intensive phase; it dominates TTFT. Decode generates tokens one by one, reusing that cache, is memory-bandwidth-intensive and dominates inter-token latency. Because their bottlenecks differ, the 2025-2026 trend is to disaggregate them: run prefill and decode on separate GPU pools and transfer the KV-cache between them. That transfer, which may travel over NVLink, over the network or through a shared storage layer, makes KV-cache movement a performance-critical operation. A factory that disaggregates prefill and decode needs a very high bandwidth, low latency KV-cache path between both pools; storage (or the DPU that governs it) stops being a passive actor and enters the critical path of every request.

The KV-cache as a performance layer

The central problem is the size of the KV-cache with long contexts: it overflows HBM, which is scarce and expensive. The options are to recompute it on every request, ruinously expensive in compute and lethal for TTFT, or to offload it to a slower layer and reuse it. Two approaches compete here. Offload to CXL memory offers gains of more than 5 times against SSD or RDMA-based caching, according to CXL supply chain vendors, and allows batches up to 30 % larger while holding latency targets, which raises throughput and GPU utilisation. Offload to NVMe, standardised by NVIDIA in 2026 with its ICMS platform on BlueField-4, is slower but much cheaper and more capacious, ideal for pod-scale KV-cache.

The big performance lever here is KV-cache reuse (prefix caching). Many requests share prefixes: the same system prompt, the same context document, the same prior conversation. If the KV-cache of those prefixes is kept in a fast storage layer and reused instead of recomputed, prefill shortens drastically and TTFT collapses. This turns a low-latency storage layer into a direct multiplier of the factory’s throughput: the more useful cache fits and the faster it is served, the less compute is wasted recalculating what has already been calculated. Sizing this layer well, in capacity, bandwidth, latency and eviction policy, is one of the most profitable performance decisions in the whole factory.

Model load time and cold start

There is a storage cost specific to inference that training ignores: starting a replica. When autoscaling decides to bring up a new instance of a model, because demand is rising, because a replica has failed or because a version is being rotated, the weights have to be read from the registry and loaded into HBM. For a model of tens or hundreds of GB, that cold start can run from seconds to minutes depending on registry bandwidth and the load path. And during that time the new GPU serves no requests while the system is, perhaps, saturated. A slow model registry translates directly into missed SLAs at peak. The mitigations, keeping weights on hot local flash, using fast-loading formats, streaming weights as they are needed, or GPUDirect Storage to load straight into HBM, are storage decisions that determine the factory’s real elasticity.

If the factory serves RAG, there is one more link in the critical path of every request: retrieval. Before the model generates anything, the system searches a vector database for the relevant fragments, and that similarity search, over indexes that can run to TB, adds to TTFT. Vector search performance depends on keeping the indexes on low-latency flash and on an efficient random read pattern; an index that does not fit in memory and is served from slow storage adds latency to every request. Beyond that, precomputed KV-caches of the retrieved documents speed up prefill, because the model reuses keys and values already calculated and only encodes the user’s query. An inference factory with RAG therefore has two latency-sensitive read workloads coexisting, the vector index and the KV-cache, competing for the same fast flash and needing to be sized together.

For an inference architect, all this introduces a storage plane with a profile of its own: latency-sensitive random reads (indexes, reusable KV-cache prefixes), high-turnover ephemeral writes (new KV-cache), large, sporadic but urgent reads (model loading) and an append-only telemetry flow. It is not sized in the least like the storage of a training cluster.

How to measure your own: beyond the spec sheet

Vendor and MLPerf figures are useful as a reference, but real performance depends on the specific workload, and an architect must measure rather than assume. The methodology that works starts from three principles.

The first is to measure accelerator utilisation, not storage in the abstract. An array can deliver 500 GB/s in a synthetic test and still leave the GPUs at 50 % because the real pattern is small random files that the sequential test does not reproduce. The right question is always: what fraction of the time is my GPU computing, with this workload, this dataset format and this pipeline?

The second is to reproduce the real I/O pattern, not a convenient one. That means testing with the same dataset format, the same shard size, the same number of workers and the same checkpointing cadence that will be used in production. MLPerf Storage’s insight was precisely to simulate accelerator think time in order to generate a realistic pattern; replicating that rigour at your own scale is what separates a useful measurement from a marketing number.

The third is to look for scaling, not peak. A platform that delivers a very high peak in a small configuration but flattens out as nodes are added is no use for growth. Tests should be run at several scales to verify that throughput grows roughly linearly and that accelerator utilisation holds, as the Hammerspace example in MLPerf v2.0 showed. A low coefficient of variation across configurations matters as much as the peak.

The cost of getting it wrong

It is worth putting a number on the problem. The price of an H100 in the cloud in mid-2026 moves across a wide range by provider, with a market median around 2.3-3.1 USD per GPU-hour. Multiply that by thousands of GPUs and by the weeks a training run lasts, and 40 % utilisation instead of 90 % translates into millions of euros of wasted compute. That is why more than 50 % of organisations report data or storage bottlenecks limiting their AI in 2026, and why storage bandwidth has emerged as a hard ceiling on scaling, comparable to power and cooling: when there is data starvation, adding compute gives diminishing returns.

The operational recommendation that keeps coming up is to size the I/O pipeline for 300 Gbps or more in serious workloads, and to measure, not assume, accelerator utilisation as a health indicator. Optimised storage can deliver up to 5 times the throughput of standard S3 access over HTTP: more than 100 GB/s aggregate against some 20 GB/s.

The physical layer: networks and PCIe Gen6

All of the above rests on a physical layer that has also advanced. NVIDIA’s ConnectX-8 NIC combines PCIe Gen6 with 800 GbE or 800 Gb InfiniBand XDR, designed for Blackwell systems. InfiniBand XDR offers 800 Gbps per direction (double NDR, which remains the usual storage fabric), and the Quantum-X800 switch adds in-network compute that accelerates collective operations. The transition to XDR is accelerating with the Blackwell deployments of 2025-2026. The mental rule is simple: storage must match the network, and the network must match the GPU. One slow link defines the performance of the whole chain.

It is worth translating this into concrete numbers. An 800 Gb/s NIC delivers, at best, around 100 GB/s to a node. If the array can serve more than the NIC absorbs, the network is the bottleneck; if the array serves less, storage is. Balance is found by sizing each layer so none strangles the next, and by verifying it with end-to-end measurements rather than the isolated specifications of each component. It is common to find installations with an excellent array and an undersized fabric, or the reverse, performing far below the sum of their parts. The physical layer also imposes its geography: network topology (how many hops there are between the compute node and the array, whether they share a switch or cross the cluster spine) affects latency as much as the array itself, and an AI storage design that ignores network topology is incomplete.

Three questions before sizing

All this complexity can be organised into three questions an architect should answer before buying anything. What is the dominant I/O pattern of my workload, large sequential, small random, bursty writes, and therefore which metric limits me? Is my workload bandwidth-limited or latency-limited, and consequently where do I invest? And what is my checkpointing budget, in size, cadence and, above all, what fraction can I overlap with compute? Whoever answers these three with measured data, rather than intuitions, sizes well; whoever skips them ends up with idle GPUs or with an oversized, very expensive array that is never put to use. AI storage performance is, at bottom, an exercise in fit: putting the right capacity where the workload needs it, no more and no less.

Takeaways

AI storage performance is not measured in GB/s but in accelerator utilisation, and that utilisation depends on three things: enough bandwidth, low latency and, above all, overlapping I/O with compute. MLPerf Storage v2.0 has finally given a common language for comparing platforms through that lens. Checkpointing is the most demanding write workload and is mastered with asynchrony, not hardware alone. Long-context inference adds a new storage layer around the KV-cache. And the cost of getting it wrong is measured in idle GPUs, which is the most expensive line in the budget.

Performance, however, is worth nothing if the data is neither protected nor available. The third article takes on the security of this new storage layer.

See also

Sources