Re:The art of high performance computing
by bayindirh on Sunday December 31, 2023 @01:39AM
(#38822289)
In our system every node gets inlet water at the same temperature via parallel piping, but when it’s in node, it goes through processors first, then RAM, then PCIe and disks. Delta T between two sockets is 5 degrees C, and the delta T between input and output is around 15-18 C depending on load.
Re:The art of high performance computing
by menaerus on Sunday December 31, 2023 @03:43AM
(#38822675)
First, thanks for sharing these details, I find them fascinating because they are not so common to be read or heard about.
> Delta T between two sockets is 5 degrees C
And secondly, ~5-10 degrees is what I see on my dual-socket workstation, and have been wondering about this delta ever since the first day I started monitoring the temperatures. At first, I thought that the heat sink wasn't installed properly but after reinstalling it the delta remained. Since I didn't notice any CPU throttling or whatsoever I figured it's "normal" and ignored it.
Re:The art of high performance computing
by bayindirh on Sunday December 31, 2023 @03:49AM
(#38822692)
Hey, no worries. Using one is equally fascinating as much as reading about it. It feels like a space shuttle, so different, yet so enjoyable.
I mean, water travels from one socket to another, so one processor adds heat equal to 5 degrees C under nominal load. The second socket doesn’t complain much, but this is enormous amounts of heat transferred in a such quick pace.
Re:The art of high performance computing
by magicalhippo on Sunday December 31, 2023 @07:00AM
(#38823465)
Interesting. What's your flow rate and pipe size?
I've seen tests done on heavy PC loops (ie multi-GPU) both high-flow and low-flow, as well as on car engines, in different coolant flow configurations. The results from all of those are that the water doesn't rise meaningfully in temperature between components.
Unless I did my back-of-the-napkin math wrong, this seems reasonable. If you have a single 10mm ID pipe going through a 1U server and up to the next, then for a full 42U rack you have about 1.7kg of water going through the servers. If the flow rate is about 1s per server (so 42 seconds for the full rack) and each 1U server dumps 500W of energy into the water, there should be just a 3 degree C difference in the water temperature between the first and the last server.