News for hackers. Stuff that matters.
Re:How to Scale Your Model: How to Think About GPUs by nickysielicki on Wednesday August 20, 2025 @12:07AM (#44958549)
The calculation under “Quiz 2: GPU nodes“ is incorrect, to the best of my knowledge. There aren’t enough ports on each GPU to saturate all the pigeonholes on the switches (less the crossbar connections), which is why 3.2TB/s of internode bandwidth is what’s offered on all of the major cloud providers and the reference systems. If it was 3.6TB/s, this would explain internode bottlenecks in distributed ring workloads.

Shamelessly: I’m open to work if anyone is hiring.

Re:How to Scale Your Model: How to Think About GPUs by aschleck on Wednesday August 20, 2025 @01:42AM (#44958961)
It's been a while since I thought about this but isn't the reason providers advertise only 3.2tbps because that's the limit of a single node's connection to the IB network? DGX is spec'ed to pair each H100 with a Connect-X 7 NIC and those cap out at 400gbps. 8 gpus * 400gbps / gpu = 3.2tbps.

Quiz 2 is confusingly worded but is, iiuc, referring to intranode GPU connections rather than internode networking.

Re:How to Scale Your Model: How to Think About GPUs by charleshn on Wednesday August 20, 2025 @10:31AM (#44962320)
Yes, 450GB/s is the per GPU bandwidth in the nvlink domain. 3.2Tbps is the per-host bandwidth in the scale out IB/Ethernet domain.
Re:How to Scale Your Model: How to Think About GPUs by jacobaustin123 on Wednesday August 20, 2025 @12:09PM (#44963195)
I believe this is correct. For an H100, the 4 NVLink switches each have 64 ports supporting 25GB/s each, and each GPU uses a total of 18 ports. This gives us 450GB/s bandwidth within the node. But once you start trying to leave the node, you're limited by the per-node InfiniBand cabling, which only gives you 400GB/s out of the entire node (50GB / GPU).
Re:How to Scale Your Model: How to Think About GPUs by xtacy on Wednesday August 20, 2025 @01:06PM (#44963748)
Is it GBps (gigabytes per second) or Gbps (giga bits per second)? I see mixed usage in this comment thread so I’m left wondering what it actually is.

The article is consistent and uses Gigabytes.

Re:How to Scale Your Model: How to Think About GPUs by shaklee3 on Thursday August 21, 2025 @12:47AM (#44969111)
GBps
Re:How to Scale Your Model: How to Think About GPUs by nickysielicki on Thursday August 28, 2025 @08:06PM (#45058434)
Try running a single node all to all with sharp disabled. I don’t believe you’ll see 450GB/s.