Cooling the AI datacenter (1/4): the thermal challenge

Contents

The power series (in Spanish) followed the watt from the utility feed all the way to the chip. But there is an implacable physical law that closes the loop: almost every watt that goes into a server comes back out as heat. An inference factory does not only have to deliver megawatts to its GPUs; it has to remove those same megawatts as heat, and remove them fast enough that the silicon does not cook itself. That is the other half of physical infrastructure, and it is the half the AI era has turned upside down. This article opens the second series in the vertical: cooling.

The approach is the same one used in the power series: universal concepts, valid for any datacenter, read throughout against the question of why AI, with its 130 kW racks and its roadmap toward the megawatt, is forcing a historic transition from air to liquid.

The law that governs everything: power is heat

Start with the physical principle, because everything else follows from it. The electrical energy a server consumes does not disappear: by conservation of energy, practically all of it turns into heat that has to be evacuated. For practical purposes, the thermal power to dissipate equals the electrical power consumed. A rack that swallows 130 kW is a 130 kW radiator, the equivalent of several dozen domestic heaters switched on at once inside a single cabinet. Cooling means moving that heat from the chip to, eventually, the atmosphere, crossing a chain of media (air, water, refrigerant) as real as the electrical chain that fed it.

The ability of a fluid to carry heat obeys a simple but revealing relation: the heat a flow can remove is proportional to its mass flow rate, its heat capacity and the temperature rise it accepts,

$$\dot{Q} = \dot{m} \cdot c_p \cdot \Delta T$$

That formula alone explains why air is in trouble. Air has a low heat capacity and a minimal density, so removing a lot of heat means moving enormous quantities of it. Water, for the same volume, carries on the order of a thousand times more heat than air. When rack density spikes, sticking with air is like emptying a swimming pool with a hairdryer: technically conceivable, practically absurd.

Put numbers on it. Solving the formula above for flow rate, removing a given power with a fixed temperature rise requires \( \dot{m} = \dot{Q} / (c_p \Delta T) \). For a 50 kW rack cooled by air with a reasonable rise of about 11 °C (the usual 20 °F), the required airflow is around 7,800 cubic feet per minute: literally a gale blowing through the cabinet. Doubling density to 100 kW doubles the gale, and there is no way to contain that much moving air without the turbulence, noise and recirculation that ruin efficiency. Water does the same job with a ridiculously smaller flow and volume, because its \( \rho \cdot c_p \) product, the capacity to carry heat per unit of volume, is overwhelmingly higher. Physics leaves no room for nostalgia: above a certain density, air simply cannot.

AI density breaks the model

For decades, air was more than enough. A traditional rack dissipated between 5 and 15 kW, and a well-designed room cooled it without breaking a sweat. Air is comfortable: cheap, safe, no pipes to drip onto the electronics. The problem is that AI has multiplied density by ten.

The root of it is the chip. An H100 (Hopper) GPU dissipated about 700 W; a B200 (Blackwell) goes up to 1,000 W, and as far as 1,200 W in its full-specification version. That increase in power per chip is the first cause of the whole problem: it is not only that there are more chips, it is that each one dissipates almost twice what the previous generation did, on a silicon area that barely grows, so heat density per square centimetre spikes. An air heatsink has a physical limit to how much heat it can pull off a small surface before the surrounding air saturates; past that point, no matter how many fans are added, the chip throttles thermally and drops its clock so as not to burn, which means wasting very expensive silicon. Liquid, with direct contact and far greater capacity, pulls that heat away without the chip having to back off.

And those chips do not travel alone: the NVIDIA GB200 NVL72 packs 72 GPUs and 36 CPUs into a single rack rated at 120 kW, with real deployments measuring 130-132 kW at full load. The roadmap does not let up: the Vera Rubin generation points at 190-230 kW per rack, Rubin Ultra “Kyber” at around 600 kW in 2027, and there is already talk of the 1 MW rack toward 2028. The distant figures should be taken for what they are, vendor roadmap rather than deployed installations, but the slope is unmistakable.

Where is the limit of air? Industry figures place it with fair consensus: air is efficient up to about 10-20 kW per rack, room-level cooling reaches about 27 kW with effort, and row-level cooling about 30 kW. With advanced techniques such as rear-door heat exchangers it can be stretched to 40-60 kW. Above that, thermodynamics stops cooperating: cooling a 50 kW rack with air would take on the order of 7,800 cubic feet per minute, a gale impossible to contain inside a cabinet. Between 50 and 100 kW, liquid stops being an option and becomes the only viable physics. AI racks sit well above that boundary. That is why 2025-2026 is the moment of the great transition.

The spectrum of solutions

It helps to see cooling not as one technology but as a spectrum ordered by density. As heat per rack rises, you climb from one rung to the next:

MethodTypical density supportedMedium that touches the heat
Room air (CRAC/CRAH) + containmentup to ~30 kW/rackAir
Rear-door heat exchanger (RDHx)up to ~40-60 kW/rackAir → water
Direct liquid to chip (DLC)~50 kW to >130 kW/rackWater/refrigerant in a cold plate
Immersion100-250+ kW/rackDielectric fluid

Each rung brings the coolant closer to the hot spot. Air cools the whole room; the rear door intercepts hot air at the rack outlet; direct liquid to chip brings water to a cold plate pressed against the processor itself; and immersion submerges the entire electronics in a fluid that does not conduct electricity. The closer the liquid gets to the chip, the more heat is captured and the higher the density that can be operated. The next three articles in this series walk that spectrum in detail: air and its limits, direct liquid to chip, and immersion.

The reality of 2026, though, is not a clean replacement but coexistence. Even a direct-liquid rack still evacuates a fraction of its heat through air, between 2 % and 20 % depending on how close the liquid gets to each component, so the hybrid room, liquid for the GPUs and air for the rest, is the norm during the transition. Vendors have understood this: the most widely deployed solutions combine both worlds.

The journey of heat, end to end

Just as the power series followed the watt from the utility feed to the chip, it is worth following the heat in the opposite direction, from chip to atmosphere, because cooling is a chain with as many links as the electrical one. Heat is born in the transistor and conducts to the chip package; from there it passes to a heatsink or a cold plate; that first medium, air or liquid, carries it to a heat exchanger; a second loop, the facility water loop, takes it out of the room; and a final rejection stage, a cooling tower, a dry cooler or a chiller, hands it to the atmosphere. Every hop has a temperature difference and a loss, and the art of cooling consists of minimising the number of hops and maximising the temperature at which you work, so that less energy (and less water) is needed for the final rejection.

Here a concept appears that runs through the whole series: the coolant distribution unit (CDU), the thermal equivalent of a transformer. The CDU separates the loop that touches the equipment, clean and controlled at precise flow and temperature, from the facility loop, and exchanges heat between the two without mixing them. It is the boundary between the server vendor’s world and the building operator’s, and where a good part of the efficiency is decided. We will look at it in detail when we get to direct liquid to chip.

The metrics of heat

To govern cooling you have to measure it, and several metrics belong in an architect’s toolkit. The first is already familiar from the power series: PUE (Power Usage Effectiveness), the ratio between total facility energy and the energy that reaches IT equipment. What matters for cooling is that cooling and air movement consume between 30 % and 45 % of a traditional datacenter’s energy: by a wide margin, the largest energy overhead on top of compute. The industry average has been stuck around 1.54 for years, according to the Uptime Institute, precisely because air cooling is inefficient and hard to improve on.

The second metric, which AI has pushed to the foreground, is WUE (Water Usage Effectiveness), the litres of water consumed per kWh of IT:

$$\text{WUE} = \frac{\text{litres of water}}{\text{kWh of IT}}$$

The average sits around 1.8-1.9 litres per kWh, although the most efficient facilities aim to go below 0.2. Water is the hidden face of cooling, and we will come back to it.

The third is the ASHRAE TC9.9 framework, the reference that defines the temperature at which equipment may operate. For air, its classes run from A1 (15-32 °C inlet) to A4 (up to 45 °C); for liquid, a recent update defines water classes W17 to W45, where the number is the maximum supply temperature in degrees. The underlying trend is clear and counterintuitive: raise operating temperatures. The hotter the coolant can come in, the less it has to be chilled artificially, and the more ambient air or water can be used without compressors. Operating hot is operating efficiently.

That idea crystallises in the concept of warm-water cooling and the ambition to operate chillerless. The reasoning is elegant: if the chip is perfectly happy with 40 or 45 °C inlet water, there is no point spending energy producing 15 °C chilled water with compressors; a dry cooler rejecting heat to outside air, which is almost always below 45 °C, is enough. NVIDIA promotes exactly that, 45 °C inlet water (class W45) for its Vera Rubin generation, with a wide thermal rise (45 °C in, 65 °C return) that reduces pumping flow and, above all, eliminates chillers in most climates. The term deserves a caveat: “chillerless” usually means “almost chillerless”, because on the hottest days some mechanical cooling backup is still needed. But the direction is unmistakable, and it is the one that saves energy, saves water and, as we will see, allows heat reuse.

One measurement caveat to close on: PUE does not capture the consumption of the fans that live inside the server, which in an AI node is not negligible. Hence TUE (Total Usage Effectiveness), which combines PUE with the equipment’s own internal efficiency (ITUE) to give a more honest picture of how much energy actually reaches the transistors. As liquid removes server fans, the distinction gains importance: in an air-cooled AI node, internal fans can consume a far from negligible percentage of the server’s own power, energy that liquid recovers by making them unnecessary. It is one of the gains of liquid that PUE, on its own, cannot even see.

Water: the new constraint

If energy is the AI datacenter’s input constraint, water is becoming an equally hard output constraint. Many efficient cooling techniques, cooling towers and evaporative cooling among them, trade water for energy by evaporating water, and at AI scale that bill has become enormous. Estimates of AI datacenter water consumption already run into hundreds of billions of litres per year, with projections doubling toward 2030. The per-query figures in circulation, from a few millilitres to half a litre per request, should be handled with care, because they vary enormously with methodology, but the aggregate order of magnitude is real and concerning.

It is worth understanding the underlying technical dilemma, because it is a trade-off rather than a defect. A datacenter’s heat rejection equipment moves inside a triangle of energy, water and temperature. A cooling tower evaporates water to cool better: little energy, a lot of water (good PUE, bad WUE). A dry cooler evaporates nothing, so it barely uses water, but it can only cool to near ambient temperature, which demands operating with hotter water and, in warm climates, more backup energy. A mechanical chiller cools to any temperature but at a high energy cost. Raising operating temperatures, as noted above, is what allows the move from tower to dry cooler without an energy penalty: warm water and the waterless datacenter are two sides of the same coin. AI, by concentrating so much heat, forces a conscious choice inside that triangle where the choice used to be made by inertia.

And it is not only a technical problem but a social one. Competition for water with communities and agriculture has begun to stall projects: in the United States, “Project Blue” near Tucson, which expected to consume hundreds of millions of gallons a year, was cancelled after local opposition. The industry’s answer is twofold: on one side, waterless designs using dry coolers in exchange for operating with hotter water (Microsoft reports reaching a WUE of 0.30 and deploying zero-evaporation cooling); on the other, direct liquid to chip itself, which by capturing heat at a higher temperature makes it easier to reject to ambient without evaporating. The paradox of 2026 is that liquid, well designed, can use less water than the evaporative air it replaces.

Waste heat: from problem to resource

There is an optimistic twist. All that heat that costs so much to remove is, after all, energy, and reusing it rather than dumping it into the atmosphere is increasingly on the table. Waste heat reuse for district heating has gone from curiosity to regulatory obligation: the European energy efficiency directive pushes in that direction, and national laws already set quotas, with Germany requiring 10 % reuse in 2026 and up to 30 % in 2028.

Here liquid again has a decisive advantage over air. Modern district heating networks need water at 50-70 °C, a temperature that a liquid-cooled datacenter with its wide thermal rise (45 °C inlet and 65 °C return, for example) can deliver almost directly, whereas the hot air of a traditional room is too cool and too diffuse to be useful. It is no accident that the large reuse cases, heating tens of thousands of homes in northern Europe with Microsoft, Google and Meta datacenters wired into district heating networks, rest on liquid cooling. For an operator, waste heat can go from cost to revenue: heat delivered to a district network can be worth considerably less than the gas it displaces, with payback periods under a decade when the datacenter sits one or two kilometres from the network. At continental scale the potential is large: European datacenter waste heat is estimated to be able to cover around 10 % of EU space heating by 2030. What was vented to the atmosphere for decades is starting to be seen for what it is: wasted high-quality energy.

What is holding the transition back

If liquid is thermodynamically inevitable, why has it not swept air aside already? Because changing cooling paradigm touches everything an operator fears touching. The first brake is fear of leaks: bringing water within centimetres of the most expensive electronics on the planet makes anyone nervous, and although modern solutions use dripless quick connectors, leak detection and drip trays, the perception of risk weighs. The second is lack of standardisation: for years every vendor brought its own fluid, its own connectors and its own temperatures, which locked customers into a supplier and complicated maintenance; the sector is converging, but slowly. The third is cost and construction: liquid demands piping, CDUs, manifolds and often a rebuilt room, an investment that only high densities justify. And the fourth is operational and cultural: a team used to managing air conditioning has to learn to operate a hydraulic system, with its water chemistry, its pumps and its gaskets.

The result is that air does not disappear: it is reserved for loads that need nothing more, while liquid enters where density forces it. The question for an architect is rarely “air or liquid?” in the abstract, but “which part of my room needs liquid, when, and how does it coexist with air everywhere else?”. Retrofit, adapting existing air rooms to host liquid islands, is one of the big businesses of 2025-2026, precisely because almost nobody starts from scratch.

Where the market is in 2026

How far along is the transition? Uptime Institute data from 2025 gives a nuanced picture: direct liquid to chip is already present in around 22 % of operators, but perimeter air remains the majority at 75 %, and 61 % of those not yet using liquid are considering it. The barriers they cite are revealing, being lack of standards, cost and fear of a leak failing over the electronics, the same ones that hold back any paradigm change. The liquid cooling market, however, is growing at high double-digit rates, with estimates multiplying it by five or six over a decade, and the large players are repositioning fast: Vertiv leads with a good share of the first GB200 racks and has launched solutions combining direct liquid to chip and air in a single module; Schneider Electric bought Motivair to enter in earnest; and alongside them live cold-plate specialists (CoolIT, Boyd, JetCool, Asetek) and immersion specialists (Submer, GRC, LiquidStack). The dominant pattern is hybrid: rooms combining liquid for the GPUs and air for everything else, as a bridge to a mostly liquid future. That vendors package both worlds into a single product says a great deal about where the transition is heading: not to a clean cut, but to a managed coexistence lasting years.

For an inference factory

What should someone building an inference factory take from this? Three ideas that run through the whole series. First, that cooling is not an add-on but a co-design: the density of AI racks forces the cooling method to be decided at the same time as the power method, because a 130 kW rack cannot simply be dropped into a room designed for air. Second, that the move to liquid is no longer optional above a certain density; the question is not whether, but which, direct to chip or immersion, and how to manage coexistence with air for years. And third, that the metrics that matter have widened: watching PUE is no longer enough, water (WUE) has to be watched too and, increasingly, what is done with waste heat, because operating cost, the social licence to build and, in some geographies, legal compliance are all decided there.

It is also worth keeping in mind that cooling is not only efficiency: it is availability. As the power series showed, a thermal failure can bring a factory down as fast as a power cut, in fact faster, because a 130 kW rack without cooling reaches dangerous temperatures in seconds. That is why critical cooling is designed with the same philosophy as power: redundancy in pumps and CDUs, early detection, and testing. A cooling system nobody has exercised at full load is, once again, an unverified promise.

This article has drawn the map of the thermal challenge. The next three go down into the detail of each rung of the spectrum: starting with air cooling and its limits, the point everyone is coming from, moving on to the direct liquid to chip that is feeding GB200 racks, and finishing with immersion and the horizon of the megawatt rack. Feeding the GPUs was half the battle; the other half is keeping them from melting.

See also

Sources