News for hackers. Stuff that matters.

A comment on Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?. Parent comment

Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by neko_ranger on Friday August 08, 2025 @04:05PM (#44841103)
Four H100 in a 2U rack didn't sound impressive, but that is accurate:

>A typical 1U or 2U server can accommodate 2-4 H100 PCIe GPUs, depending on the chassis design.

>In a 42U rack with 20x 2U servers (allowing space for switches and PDU), you could fit approximately 40-80 H100 PCIe GPUs.

[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by jzymbaluk on Friday August 08, 2025 @04:17PM (#44841245)
And the big hyperscaler cloud providers are building city-block sized data centers stuffed to the gills with these racks as far as the eye can see
[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by michaelt on Friday August 08, 2025 @05:02PM (#44841654)
Why stop at 80 H100s for a mere 6.4 terabytes of GPU memory?

Supermicro will sell you a full rack loaded with servers [1] providing 13.4 TB of GPU memory.

And with 132kW of power output, you can heat an olympic-sized swimming pool by 1°C every day with that rack alone. That's almost as much power consumption as 10 mid-sized cars cruising at 50 mph.

[1] https://www.supermicro.com/en/products/system/gpu/48u/srs-gb...

[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by handfuloflight on Friday August 08, 2025 @07:26PM (#44842739)
What about https://www.cerebras.ai/system?
[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by procaryote on Saturday August 09, 2025 @04:11AM (#44844860)
> as much power consumption as 10 mid-sized cars cruising at 50 mph

Imperial units are so weird

[ Parent | on HN ]