Thanks for the links!

The first one says that networking infrastructure costs 13% and the other says 1.5%. Not a great start.

Let's imagine that a traditional full rack can host 80ish H100s and that each one costs $25k. That's 2 million right there. There's probably only 10x cases in that rack so that's 10x dual core motherboards, RAM, network, etc. Call them 20k each. Maybe you need a 32 port 100gig fiber switch, that's 8k. Let's say it's 2.3 million per rack and 2 million of it is just the GPUs. We're at 85% right there.

I suspect that it would take more than just another 100k per rack to finish out a data center so my "80% of the cost is in the GPUs" is definitely wrong. But I didn't see anything in these articles that indicated to me that 50% of the cost of a datacenter is anything really durable like buildings or electrical infrastructure or cooling. There's still an enormous amount of cost in stuff whose performance follows more of an exponential performance curve, not linear. It's just an absolute nightmare to try and amortize exponential performance curves with linear repayment.

These days there's usually two networks. Multi-trillion parameter models are too big to fit on one box of 8x H100's, so many of these are running RDMA (800Gb/sec on GB300 series) for rack-scale/many row-scale inference.

For GB300 series, pricing along the lines of $500k/server, 9 servers/rack ($4.5M/rack) before looking at water cooling, 800Gb RDMA network, and your 100Gb IP network