Storage in the AI era (4/4): availability

Contents

We have covered the state of the art, performance and security. The series closes with the property all the others depend on: availability. Storage that is blazingly fast and perfectly encrypted is worth nothing if a failure brings down an inference factory under SLA, halts a training run across thousands of GPUs or lets a petabyte-scale dataset corrupt itself in silence. And at AI scale, failures are not an exceptional event: they are the permanent operating regime. The difference lies in how you live with them: training resumes from a checkpoint; an inference factory, by contrast, cannot afford to stop, and must keep answering while a component fails underneath it. We will look at both cases, with the focus on continuous service.

Availability and durability are not the same thing

It is worth starting by separating two concepts that often get confused. Availability measures what fraction of the time the data is reachable; durability measures the probability of not losing it. A system can be very durable but poorly available (the data is safe but cannot be read right now) and vice versa.

Availability is modelled with the familiar relation between mean time between failures and mean time to repair:

$$A = \frac{\text{MTBF}}{\text{MTBF} + \text{MTTR}}$$

From that come the “nines” everyone quotes, whose translation into annual downtime is worth internalising:

Availability“Nines”Annual downtime
99.9 %three nines~8.8 hours
99.99 %four nines~52.6 minutes
99.999 %five nines~5.3 minutes
99.9999 %six nines~32 seconds

One detail that surprises anyone coming from the on-premise world: public cloud storage services do not usually offer an SLA above four nines (99.99 %). AWS S3, for instance, guarantees 99.99 % availability per region. Six nines are the territory of enterprise arrays in a stretch cluster configuration: Pure Storage offers them in that topology, against “five nines and a bit” on a single node. The formula makes it clear why reducing MTTR, recovering fast, is usually more cost-effective than chasing an impossible MTBF.

Durability: the “eleven nines” and what they hide

Durability is expressed in figures that look like marketing but have a precise meaning. The “eleven nines” of S3 (99.999999999 %) mean that, if you store ten million objects, you would expect to lose one every ten thousand years. How do you get there? AWS starts from the unrecoverable failure rates of the hardware, replicates across three independent availability zones per region and applies erasure coding with overlapping fragmentation. The honest caveat is that there is no agreed standard for how the nines are calculated: every vendor uses its own methodology, and comparing durability figures across providers is, strictly speaking, comparing assumptions.

The mechanism that produces those stratospheric numbers is the combination of redundancy and failure independence. If a system stores each object so that it tolerates the loss of several fragments, and distributes those fragments across independent failure domains (different drives, nodes, racks, availability zones), the probability of simultaneously losing all the fragments you need becomes astronomically small. The key word is independent: if the fragments share a single point of failure, the same power supply, the same switch, the same zone, the independence is fictitious and the calculated durability an illusion. A good share of real data losses in theoretically durable systems come from failure correlations the model did not contemplate: a batch of faulty drives from the same manufacturer, a software bug that corrupts every replica at once, an accidental deletion that propagates. Real durability demands diversity, not just quantity.

To anchor those assumptions in reality there is one invaluable source: Backblaze’s drive statistics. In its 2025 data (published in February 2026) the annualised failure rate (AFR) of its fleet of 344,196 drives fell to 1.36 %, down from 1.55 % in 2024, the lowest since 2022; the lifetime AFR remains stable at 1.30 %. Drives of 14-16 TB now make up 52 % of the fleet and those of 20 TB or more around 23 %. These numbers are the raw material from which any credible durability figure is calculated.

But total drive failure is not the most insidious threat. That would be silent corruption (bit rot). Enterprise drives have an unrecoverable read error rate on the order of one bit per \( 10^{14} \) bits read, roughly 11 TB, and that rate has barely improved while capacity has soared. A classic NetApp study of 1.53 million drives over 41 months found more than 400,000 checksum mismatches, of which 30,000 were not detected by the RAID controller and only appeared during scrubbing; nearline drives corrupt at ten times the rate of enterprise ones. The lesson: at petabyte scale, periodic scrubbing and end-to-end checksums (such as those of ZFS) are not optional, because the corruption RAID does not see is the one that destroys a training dataset without warning.

Redundancy: why classic RAID is no longer enough

The traditional way of tolerating drive failures, RAID with parity, has become problematic precisely because of capacity growth. The reason is rebuild time. Sequential write speed on drives has not grown at the pace of their capacity, so rebuilding a large drive takes longer and longer: a 16 TB nearline HDD in RAID 5 can take between 24 and 72 hours to rebuild. During that window the array is degraded and exposed to a second failure.

And there is a subtler risk. Rebuilding a drive forces you to read all the others in full, which multiplies the exposure to an unrecoverable read error. Rebuilding a RAID 5 of eight 8 TB drives means reading 56 TB; with enterprise drives that implies around a 30 % probability of hitting a URE that aborts the rebuild, and with consumer drives the probability approaches 96 %. That is why RAID 5 is, quite simply, deprecated for new arrays with drives larger than 4 TB. On top of that comes the write hole: if a power cut interrupts a write, data and parity can end up out of sync and corrupt the array. ZFS eliminates it with copy-on-write.

The modern answer has two pieces. The first is erasure coding (Reed-Solomon codes, \( k+m \) schemes): the object is split into \( k \) data fragments and \( m \) parity fragments, and any \( k \) of the \( k+m \) are enough to reconstruct it. Its efficiency against replication is notable. Storage overhead is:

$$O = \frac{k+m}{k}$$

A \( k=4 \), \( m=2 \) scheme gives an overhead of 1.5, so 1 TB logical occupies 1.5 TB physical and tolerates two simultaneous failures, against the factor of 3 of triple replication. At exabyte scale, that difference between 1.5 and 3 is decisive in cost. The price of erasure coding is more CPU on writes and rebuilds and more complex operations, which is why replication is still preferred for latency-sensitive data read locally.

The second piece is declustered RAID (dRAID): instead of concentrating the rebuild on a dedicated spare drive, it distributes rebuild I/O across all the drives in the pool. The effect on recovery time is drastic: a 20 TB drive that in traditional RAID would take more than 60 hours can resilver in under 15 because every surviving drive contributes at once. OpenZFS dRAID and IBM Storage Scale implement it. The underlying idea connects back to the availability formula: if you cannot avoid the failure, minimise MTTR by spreading the recovery work.

There is one variable that amplifies all of this and that flash has changed at the root: the speed of the medium during the rebuild. Rebuilding 10 TB on HDD, whose transfer rate is around 300 MB/s, can take more than 55 hours; on SSD, with transfers above 6,000 MB/s, it drops to under 3, twenty times faster. When flash is combined with dRAID, the vulnerability window after a failure shrinks from days to hours or minutes, and with it the probability of a second concurrent failure causing data loss. This is one of the underlying reasons why all-flash platforms have taken over in AI: they do not just perform better, they recover sooner, and at this scale recovery speed is data safety.

Replication and RPO/RTO: how much you can lose and how long you can wait

To tolerate the failure of an entire site, replication comes into play along with two metrics every continuity plan must set: the RPO (Recovery Point Objective, the maximum acceptable data loss, measured in time) and the RTO (Recovery Time Objective, the maximum acceptable downtime).

Synchronous replication mirrors each write to a second site before acknowledging it, guaranteeing an RPO of zero, zero data loss, but its reach is limited by latency: in practice it requires metropolitan links below about 10 ms, and it does not work for intercontinental distances. Asynchronous replication acknowledges the write locally and propagates afterwards, which allows any geographic distance in exchange for an RPO of minutes or hours. Active-active stretch cluster topologies (such as Pure ActiveCluster) achieve zero RPO and RTO within the metropolitan range, and some add a third asynchronous site to combine local consistency with geographic protection. The decision is always the same triangle: distance, cost and tolerable loss. You cannot optimise all three at once.

It is worth dismantling a myth about array high availability. People tend to assume that “active-active” is always superior to “active-passive”, but there is a performance nuance that matters. In an active-passive configuration, the secondary controller takes over the load after a failure without having been serving I/O, so post-failover performance is predictable. In active-active, both controllers serve load in normal operation; when one fails, the survivor has to absorb its own load plus the transferred one, and may miss the performance SLA exactly when it is needed most. There is no universal answer: the choice depends on whether you prioritise aggregate performance in normal operation or a performance guarantee in degraded mode. What is non-negotiable is non-disruptive upgrades (NDU): a serious AI array is upgraded with the system powered on and serving I/O, using the same dynamic failover routine it employs on a failure. If an upgrade requires stopping, that array is not up to a critical workload.

The permanent failure regime: why it matters to inference

This is where AI storage availability separates from everything above: at AI scale, hardware failures are not rare, they are continuous. The most quoted figure comes from Meta’s Llama 3 training on a cluster of 16,384 H100 GPUs: over 54 days it suffered 419 unexpected component failures, one failure every three hours on average, with a cluster MTBF of approximately 1.8 hours. More than 66 % of the interruptions were due to hardware faults: defective GPUs, HBM3 memory, SRAM, network switches and cables.

Although the figure comes from a training run, the underlying statistic applies equally to an inference factory: the same hardware fails at the same rate, and an inference fleet of hundreds or thousands of GPUs will see components drop daily. The difference lies in the response. Training absorbs the failure by resuming from a checkpoint; inference absorbs it with redundancy and hot switching, because it cannot stop. In both cases storage is the piece that makes recovery possible: in training, as the medium where the checkpoint lives; in inference, as the replicated repository from which a replica is stood up instantly to substitute the fallen one. That is why it pays to understand the checkpointing mechanism well even if the factory is an inference one: it is the same principle of “assume the failure and recover fast” taken to its most extreme form.

The reason is purely statistical. If an individual component has a given MTBF, the MTBF of a cluster of \( N \) identical components shrinks in inverse proportion:

$$\text{MTBF}_{\text{cl}} = \frac{\text{MTBF}_{\text{dev}}}{N}$$

Even if each GPU failed only once every six years (some 50,000 hours), a cluster of 100,000 GPUs would see a failure every half hour, and one of a million, every three minutes. This is the operational reality no AI storage design can ignore.

How do you live with this? With checkpointing, which is both a performance tool (we saw it in article 2) and the central resilience mechanism: after a failure, training resumes from the last checkpoint instead of starting from scratch, potentially saving weeks of compute. This inverts the usual relationship: storage is not only what has to be protected, it is the instrument that protects the GPU. There is even an underlying debate, with Epoch AI arguing that with checkpointing based on memory distributed across GPUs, rather than storage-based, hardware failures would stop limiting training size even beyond a million GPUs. Whatever the approach, the conclusion for the architect is that the availability of the checkpointing subsystem is the availability of the training run.

A revealing detail in Meta’s data is that more than two thirds of the interruptions were hardware (GPU, HBM3, SRAM, switches and cables), and that storage, when well designed, does not appear among the main causes of failure but among the recovery mechanisms. This inverts the usual psychology of the storage architect: at this scale the job is not so much about preventing your subsystem from failing, though it must, as about guaranteeing that, when anything else fails, training can resume quickly. The storage of an AI cluster is, to a large extent, resilience infrastructure for the rest of the cluster. Sizing the write bandwidth for checkpointing and the re-read speed after a restart is therefore an availability decision for the whole system, not an isolated array metric.

Availability in an inference factory: the service does not stop

Everything above (failures at scale, checkpointing, cluster MTBF) is described from the point of view of training, which is a batch job: if it goes down, it resumes and time is lost, but nobody is waiting on the other side. An inference factory is the opposite: an online service under an SLA, where every second of unavailability is a failed request and, often, a contractual penalty. Its availability is designed with a different mindset.

The first principle is that the service plane must survive the failure of any component without interrupting requests. That means redundant replicas of every served model, load balancing that automatically withdraws a fallen replica, and, in the storage plane, that none of the factory’s four planes is a single point of failure. The model repository must be replicated and available, because if it goes down you cannot start new replicas exactly when you need them to substitute the failing ones. The RAG vector database needs its own high availability, because without it the service degrades its quality or stops answering. And the KV cache layer, ephemeral as it is, conditions latency: losing a shared cache forces recomputation and spikes TTFT at once for every affected client.

The second principle is model updates with no downtime. In an inference factory models are rotated frequently, with new versions, adjustments and rollbacks, and doing so without cutting service demands progressive deployment patterns: blue-green (stand up the new version in parallel and switch traffic when it is warm) or canary (send a fraction of traffic to the new version and widen it if it behaves). Both patterns have a direct storage implication: during the transition two sets of weights coexist in the repository and, at times, in HBM, which demands capacity and load bandwidth so the new version is ready before it receives traffic. A slow model repository does not just delay autoscaling: it slows every deployment and lengthens the risk window.

The third principle is elastic autoscaling, which turns cold start time, the time to load weights from storage that we saw in the performance article, into an availability metric. If starting a new replica takes minutes because the repository is slow, the factory does not absorb the peaks and degrades or rejects requests precisely when demand is highest. The real elasticity of an inference factory is limited by the speed at which its storage delivers models. That is why many installations keep a reserve of weights in hot local flash and use accelerated load paths: availability under variable load is won, in large part, in the storage layer.

The fourth principle is redundancy of the service plane itself, which can be sized with numbers. If a service replica has an individual availability \( a \) (limited by its GPU, its node and its access to storage), a service with \( n \) independent replicas in which one is enough to answer reaches an availability of:

$$A = 1 - (1 - a)^{n}$$

The formula explains why redundancy is so effective: with reasonably available replicas, each additional replica adds nines to the service. But its hypothesis is the same one we saw in durability: independence. If every replica depends on the same model repository, the same vector database or the same KV cache layer, that shared component is the real limit on availability no matter how many GPU replicas you add. Hence, in an inference factory, replicating and giving high availability to the storage planes (repository, vector DB, cache) is as important as replicating the serving GPUs themselves: those, and not the compute, are the single points of failure that usually get overlooked.

The conclusion is that in an inference factory the availability of storage is measured in terms of continuous service: not “how long do I take to recover the data?” but “does the service keep answering while a component goes down, a model is updated or demand doubles?”. It is a target of five or six nines over a living system, not durability over an archive at rest.

Disaster recovery: the 3-2-1 rule and the petabyte problem

For the major disaster, loss of a site or a destructive attack, the reference is still the 3-2-1 rule: three copies of the data, on two different media types, with at least one off site. Its 2025 evolution adds two requirements: immutability (the off-site copy must be unalterable, connecting with the anti-ransomware defence of the security article) and verified recoverability (a backup you have not tested restoring is not a backup).

At AI scale a physical obstacle appears that is hard to dodge: the recovery time of massive datasets. Restoring petabytes from an off-site copy can take days or weeks over standard connections, an unacceptable RTO for many operations. Organisations with petabytes cannot even do the initial backup to cloud over conventional internet; they need physical transfer or dedicated links. Sizing the recovery, not just the copy, is where most AI DR plans fail.

When the cloud goes down too

It is worth remembering that delegating to a hyperscaler does not eliminate the problem, it only moves it. 2025 left a forceful reminder: the big AWS outage of 19-20 October 2025, caused by a race condition in an internal DynamoDB microservice that generated an empty DNS record, cascaded to EC2 and lasted some fifteen hours, with more than seventeen million reports and global services down. There were precedents in July 2024 (us-east-1, via Kinesis) and February 2025 (Stockholm). The moral for a critical AI architecture is the usual one: multi-region or multi-cloud resilience is a conscious design decision, not a property you inherit by signing up for cloud.

The trend: storage that heals itself

The evolution vector for 2026 is the incorporation of AI into the storage layer itself for self-healing defence: predicting failures before they happen, automatic tiering and migration, pre-emptive remediation and instant recovery from ransomware. Storage hardware “is no longer passive”, as we saw in security. At exabyte scale, object storage is consolidating as the core because it combines massive durability with flexible metadata and versioning, while tape re-emerges as an active archive tier for AI datasets, fighting the cost of flash and HDD. With data growth of 30-40 % a year, automating resilience stops being a convenience and becomes a necessity: at that scale no human team manages failures by hand.

Commercial guarantees have moved in the same direction. Some vendors already offer contractual SLAs that were previously unthinkable: Pure, with its Evergreen//One model, announces up to ten distinct guaranteed SLAs, including six nines of uptime, zero data loss, a capacity buffer and zero planned downtime for upgrades, plus a specific post-cyberattack recovery SLA that guarantees clean arrays. Whatever the vendor, the trend signals a change of model: from selling hardware to guaranteeing availability outcomes. For the architect, this shifts part of the risk onto the provider, but it does not exempt you from designing: a contractual SLA compensates financially for an outage, it does not prevent it, and the continuity of a training run worth millions of euros is not restored with a credit note.

Availability is governed with observability

None of these techniques works blind. Sustaining availability at AI scale demands observability: monitoring the state of every drive (with SMART-style telemetry and failure prediction), the progress and integrity of scrubs, replication latency against its RPO target, and the health of checkpoints, which are written, yes, but which also have to be re-readable. The most deceptive metric is the backup that has never been restored: an unverified copy is a hypothesis, not a guarantee. The operational discipline that distinguishes resilient infrastructure is the periodic recovery test, at realistic scale, measuring real RTO against the target. At the scale of petabytes and thousands of GPUs, what is not measured cannot be guaranteed, and what is not rehearsed tends to fail precisely on the day it is needed.

Takeaways, and closing the series

The availability of AI storage is designed on four pillars: efficient redundancy (erasure coding and dRAID instead of classic RAID, which no longer copes with large drives), verified durability (with scrubbing and checksums against silent corruption), replication with explicit RPO/RTO according to criticality, and a pipeline resilience that assumes failure as the normal state and leans on checkpointing. The availability formula is a reminder of the practical priority: when you cannot avoid the failure, and at AI scale you cannot, what makes the difference is recovering fast.

With this we close the series. The state of the art gave us the map; performance, the reason for being; security, the protection; and availability, the guarantee that everything above will still be there tomorrow. In the AI era, storage has stopped being the silent basement of the datacenter and become what decides whether the GPUs, the most expensive and scarcest resource, work or wait.

See also

Sources