News for hackers. Stuff that matters.
Re:The art of high performance computing by bayindirh on Saturday December 30, 2023 @06:33PM (#38820363)
Because it’s complex. Even more complex than “engineered” air.

You need two circuits, and a CDU between them. Coolants needs maintaining. You add antifreeze, biocides, etc.

Air is brute force. It cools everything it touches. Liquid cooling is serialized in a node. Two sockets? Second will be hotter. HBA not making good contact? It’ll overheat.

You add extensive leak detection subsystems, the amount of coolant moving in your primary circuit becomes massive.

Currently you can remove 97% of the heat via liquid (including the PSUs), and it’s cheaper to do so than air, but it’s not “rails, screws, cables, power on”. Air cooled systems can be turned on in a week. Liquid cooled ones take a month.

However, using liquid is mandatory after some point. You can’t cool systems that dense and under that load with air. They’ll melt.

Re:The art of high performance computing by gopher_space on Saturday December 30, 2023 @08:12PM (#38820916)
What's this all look like without an atmosphere?
Re:The art of high performance computing by eutropia on Saturday December 30, 2023 @08:45PM (#38821048)
Worse, heat dissipation is a major constraint for spacecraft and satellites because you can only radiate heat away as infrared photons.
Re:The art of high performance computing by uticus on Saturday December 30, 2023 @11:19PM (#38821759)
Amazing considering how much heat travels from Sun (and punches through atmosphere) to Earth surface. Didn’t realize there was that much of an insulation property.
Re:The art of high performance computing by pixelpoet on Sunday December 31, 2023 @05:29AM (#38823027)
Besides the heat insulation, without the vacuum of space we would all be deafened by the sun's roar.
Re:The art of high performance computing by mr_toad on Sunday December 31, 2023 @10:06PM (#38829472)
Vacuum is a great insulator, which is why we have vacuum flasks.
Re:The art of high performance computing by taneq on Saturday December 30, 2023 @09:57PM (#38821350)
Doesn’t have to be infrared but yeah, space isn’t “cold” so much as it’s an insulator.
Re:The art of high performance computing by _a_a_a_ on Sunday December 31, 2023 @07:48AM (#38823718)
at ~2 kelvin I'd have thought you can radiate away a truckload of heat surely
Re:The art of high performance computing by zmgsabst on Sunday December 31, 2023 @08:43AM (#38823982)
You can radiate easily.

But not convect. Hence why it’s much, much harder than removing heat on Earth.

Re:The art of high performance computing by _a_a_a_ on Sunday December 31, 2023 @09:10AM (#38824135)
Of course you're not convecting but if you are radiating from a hot body into an ambient two Kelvin then you are going to lose heat really, really fast. IIRC heat loss by black body radiation into its surroundings is proportional to the fourth power of the temperature difference between the body and surroundings (from memory, and going back a very long way, so maybe incorrect).
Re:The art of high performance computing by amluto on Sunday December 31, 2023 @03:18PM (#38827172)
> proportional to the fourth power of the temperature difference between the body and surroundings

Almost. It’s proportional [0] to T_hot^4 - T_cold^4. For a 100C surface with emissivity 1, that’s about 1kW/m2 if there is no radiation coming back, which really isn’t very high. You cannot cheat this with fancy folded-up radiating surfaces (it’s thermodynamically impossible, and the actual mechanism that kills it is one fin of the heatsink radiating right at the next one).

So cooling in space is hard. You’re not getting GPU-like power densities without a physically immense radiating surface extending way past those GPUs.

[0] Caveat: emissivity can depend on wavelength, and the law holds independently at each wavelength. So this can introduce interesting effects, which is how all the fancy prototype roof-cooling materials work, and it’s also related to how “spectrally selective” windows and window films work.

Re:The art of high performance computing by _a_a_a_ on Monday January 01, 2024 @05:17PM (#38835840)
Thanks, you clearly know your subject. I must say that without doubting your figures, I am astonished at how little heat can be lost radiatively. Given the 4th powers I assumed it must be vastly higher, but clearly not, much against my intuition. Thanks for a good answer!
Re:The art of high performance computing by fecal_henge on Sunday December 31, 2023 @04:10AM (#38822769)
You just need to radiate in the visible spectrum then the problem will be much reduced.
Re:The art of high performance computing by gopher_space on Tuesday January 02, 2024 @07:24PM (#38849095)
Couldn't you heat up and poop out blocks of material all day long, or is that starting down a "solution creates more heat than it dissipates" path?
Re:The art of high performance computing by magicalhippo on Sunday December 31, 2023 @12:50AM (#38822126)
> Liquid cooling is serialized in a node. Two sockets?

I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components.

Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe going through a 1U server and up to the next, then for a full 42U rack you have about 1.7kg of water going through the servers. If the flow rate is about 1s per server (so 42 seconds for the full rack) and each 1U server dumps 500W of energy into the water, there should be just a 3 degree C difference in the water temperature between the first and the last server.

Re:The art of high performance computing by bayindirh on Sunday December 31, 2023 @01:39AM (#38822289)
In our system every node gets inlet water at the same temperature via parallel piping, but when it’s in node, it goes through processors first, then RAM, then PCIe and disks. Delta T between two sockets is 5 degrees C, and the delta T between input and output is around 15-18 C depending on load.
Re:The art of high performance computing by menaerus on Sunday December 31, 2023 @03:43AM (#38822675)
First, thanks for sharing these details, I find them fascinating because they are not so common to be read or heard about.

> Delta T between two sockets is 5 degrees C

And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after reinstalling it the delta remained. Since I didn't notice any CPU throttling or whatsoever I figured it's "normal" and ignored it.

Re:The art of high performance computing by bayindirh on Sunday December 31, 2023 @03:49AM (#38822692)
Hey, no worries. Using one is equally fascinating as much as reading about it. It feels like a space shuttle, so different, yet so enjoyable.

I mean, water travels from one socket to another, so one processor adds heat equal to 5 degrees C under nominal load. The second socket doesn’t complain much, but this is enormous amounts of heat transferred in a such quick pace.

Re:The art of high performance computing by magicalhippo on Sunday December 31, 2023 @07:00AM (#38823465)
Interesting. What's your flow rate and pipe size?
Re:The art of high performance computing by is_true on Sunday December 31, 2023 @08:37AM (#38823956)
Is rising air's water content (humidity) worth it? Humid air can "store" more heat.

I guess it could be bad past some %, but there's a probably a point where it's worth it.