> It always made me wonder why liquid cooling wasn't more of a thing for datacenters.

Liquid cooling is almost a defacto-standard in data centers in the HPC world. The Top of the TOP500 machines are all liquid cooled. Not by choice, but due to physics constraints.

There is a big gap in power density between the HPC world and the usual datacenter-commodity-hardware world.

Commodity DS are designed with the assumption that the average machine will run with a fraction of it's maximum load. HPC systems at the opposite are designed to operate safely at 100% load all the time.

In a previous company where I worked, we attempted to install a medium size HPC cluster in a well-known commerical datacenter and network provider. The commercial of the DS almost felt from his chair when we announced the power requirements.

> we attempted to install a medium size HPC cluster in a well-known commerical Datacenter and network provider. The commercial of the DS almost fall from his chair when we announced the power requirements.

Heh. We tried it too. They didn’t believe that a single node used their entire rack’s budget at first.

Sounds fascinating. Can you give any more details? What kind of nodes are they and how they differ from "traditional" DC hardware, say from Supermicro?

The difference is GPUs. A normal dual socket system serving a database or webserver use under medium load around 200-300W, One of these [1] equipped with 10xA100 can easily use in the ballpark of 3kW under load. So we are talking 10x the power usage.

[1]https://www.supermicro.com/en/products/system/gpu/5u/sys-521...

Thanks, this makes sense!

I'm going through a similar thing. Next week our procurement is arriving, but most likely majority of the nodes will sit in a box for a while before they can figure out how to power them...

Also, this involves a UK university where it would takes forever to upgrade the power delivery to the building/floor/room. There's no planned upgrade whatsoever so people just need to make do with what we have.

I’m a bit skeptical about the claim that the top of the TOP500 are all liquid cooled due to physical constraints. (Where the “physics” here seems to mean the density and property of air cool in general but ignoring the environment of the machine such as the ambient weather.)

The one data point I know well is NERSC, and has been air cooled until relatively recently. Part of the success of air cool in the past was the great Bay Area weather that is typically cooled enough. I forgot the exact reason for upgrading to water cool, but it was before the recent upgrade to a new machine (Perlmutter). It may have to do with our weather getting more extreme so that there are more incidences that air cool only will “throttle” your machine.

The reason I’m skeptical about that claim is that the ambient temperature also play a role. But certainly the density is increasing so may be the biggest supercomputers at the moment are too dense (in power consumption) to be air cooled.

If you take the top5 of the top500 (01/2024):

- Frontier: Liquid cooled

- Aurora: liquid cooled

- Eagle: liquid cooled

- Fugaku: liquid cooled

- Lumi: Liquid cooled with heat recycling.

The need for liquid cooled arrived with the usage of accelerators that bumped significantly the power density.

That said, Fugaku which is not accelerator driven is also liquid cooled today.

I am surprised to learn that NERSC was air cooled until recently. Most of the BGQ machines, that were dominating the top500 10 years ago, were already liquid cooled.

I probably wasn't clear. I don't doubt your first statement "The Top of the TOP500 machines are all liquid cooled." My skepticism is your second statement "due to physics constraints". It may very well be true but I'm just skeptical that it can't be done with a combination of engineering and ambient environment (mostly temperature but also a tradeoff between density and space). May be I focused on the word physics too much (being a physicist), but it seems the decision would basically be a cost-benefit analysis and risk management which involves many factors including money, maturity of solutions in the market, safety, etc. For example, at NERSC, there are real risk of a massive earthquake long overdue (the whole floor is quake-proof, but I guess the risk is reduced, not eliminated), so I guess they probably have considered this in that design choice.

But perhaps you're right that the physics is the ultimate factor here: what comes in must goes out. With order of magnitude scale of increase in power coming in, water seems to be the most effective and cheap entity to absorb those heat and be transported outside the floor very efficiently.

Immersion cooling is getting big. At the last Supercomputing conference I probably saw at least a dozen vendors of immersion cooling equipment. My datacenter has one cluster with liquid cooling caps over the sockets, and two immersed clusters. The latter two have basins of various degrees of sophistication under them for when they do spring a leak.

Because it’s complex. Even more complex than “engineered” air.

You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc.

Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat.

You add extensive leak detection subsystems, the amount of coolant moving in your primary circuit becomes massive.

Currently you can remove 97% of the heat via liquid (including the PSUs), and it’s cheaper to do so than air, but it’s not “rails, screws, cables, power on”. Air cooled systems can be turned on in a week. Liquid cooled ones take a month.

However, using liquid is mandatory after some point. You can’t cool systems that dense and under that load with air. They’ll melt.

What's this all look like without an atmosphere?

Worse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.

Amazing considering how much heat travels from Sun (and punches through atmosphere) to Earth surface. Didn’t realize there was that much of an insulation property.

Besides the heat insulation, without the vacuum of space we would all be deafened by the sun's roar.

Vacuum is a great insulator, which is why we have vacuum flasks.

Doesn’t have to be infrared but yeah, space isn’t “cold” so much as it’s an insulator.

at ~2 kelvin I'd have thought you can radiate away a truckload of heat surely

You can radiate easily.

But not convect. Hence why it’s much, much harder than removing heat on Earth.

Of course you're not convecting but if you are radiating from a hot body into an ambient two Kelvin then you are going to lose heat really, really fast. IIRC heat loss by black body radiation into its surroundings is proportional to the fourth power of the temperature difference between the body and surroundings (from memory, and going back a very long way, so maybe incorrect).

> proportional to the fourth power of the temperature difference between the body and surroundings

Almost. It’s proportional [0] to T_hot^4 - T_cold^4. For a 100C surface with emissivity 1, that’s about 1kW/m2 if there is no radiation coming back, which really isn’t very high. You cannot cheat this with fancy folded-up radiating surfaces (it’s thermodynamically impossible, and the actual mechanism that kills it is one fin of the heatsink radiating right at the next one).

So cooling in space is hard. You’re not getting GPU-like power densities without a physically immense radiating surface extending way past those GPUs.

[0] Caveat: emissivity can depend on wavelength, and the law holds independently at each wavelength. So this can introduce interesting effects, which is how all the fancy prototype roof-cooling materials work, and it’s also related to how “spectrally selective” windows and window films work.

Thanks, you clearly know your subject. I must say that without doubting your figures, I am astonished at how little heat can be lost radiatively. Given the 4th powers I assumed it must be vastly higher, but clearly not, much against my intuition. Thanks for a good answer!

You just need to radiate in the visible spectrum then the problem will be much reduced.

Couldn't you heat up and poop out blocks of material all day long, or is that starting down a "solution creates more heat than it dissipates" path?

> Liquid cooling is serialized in a node. Two sockets?

I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components.

Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe going through a 1U server and up to the next, then for a full 42U rack you have about 1.7kg of water going through the servers. If the flow rate is about 1s per server (so 42 seconds for the full rack) and each 1U server dumps 500W of energy into the water, there should be just a 3 degree C difference in the water temperature between the first and the last server.

In our system every node gets inlet water at the same temperature via parallel piping, but when it’s in node, it goes through processors first, then RAM, then PCIe and disks. Delta T between two sockets is 5 degrees C, and the delta T between input and output is around 15-18 C depending on load.

First, thanks for sharing these details, I find them fascinating because they are not so common to be read or heard about.

> Delta T between two sockets is 5 degrees C

And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after reinstalling it the delta remained. Since I didn't notice any CPU throttling or whatsoever I figured it's "normal" and ignored it.

Hey, no worries. Using one is equally fascinating as much as reading about it. It feels like a space shuttle, so different, yet so enjoyable.

I mean, water travels from one socket to another, so one processor adds heat equal to 5 degrees C under nominal load. The second socket doesn’t complain much, but this is enormous amounts of heat transferred in a such quick pace.

Interesting. What's your flow rate and pipe size?

Is rising air's water content (humidity) worth it? Humid air can "store" more heat.

I guess it could be bad past some %, but there's a probably a point where it's worth it.

OVH prominently uses water cooling including custom components with their own design.

https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...

https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...

https://blog.ovhcloud.com/new-hybrid-immersion-liquid-coolin...

In contexts where there's a good chance of standardisation, I believe it is. Both OCP [0] and Open19 [1] have liquid cooling as part of the standard.

[0]: https://www.opencompute.org/projects/cooling-environments

[1]: https://gitlab.com/open19/v2-specification/-/blob/main/syste...

I wonder if you could design a rack or half rack that has a connector for attaching a minisplit refrigerant line directly to it, instead of cooling the entire room.

Problem is that’s how you sell 1-3 racks not an entire room of them.

Ethylene lines might be more practical, easier to wire into existing systems.