Serving AI from the Basement – 192GB of VRAM Setup

Re:Serving AI from the Basement – 192GB of VRAM Setup by bluedino on Sunday September 08, 2024 @04:07PM (#41482783)

Take your typical 'GPU node', which would be a Dell/HP/SuperMicro with 4-8 NVIDIA H100's and a single top high level AMD/Intel CPU. You would need 2-4 240v outlets (30A).

In the real world you would plug them into a PDU such as: https://www.apc.com/us/en/product/AP9571A/rack-pdu-basic-1u-...

Each GPU will take around 700W and then you have the rest of the system to power, so depending on CPU/RAM/storage...

And then you need to cool it!

[ Parent | on HN ]

Re:Serving AI from the Basement – 192GB of VRAM Setup by XMasterrrr on Monday September 09, 2024 @02:22AM (#41485934)

Yeah, you get it. I have two PDUs with NEMA L6-30P plugin, and two NEMA L6-30R outlet. Each outlet is connected to it's dedicated 30A 240v circuit (which gives you a total of 60A, or 6000w).

Cooling is its own story... Man, figuring out this process was hella of a journey

[ Parent | on HN ]