Re:Tesla's TTPoE at Hot Chips 2024: Replacing TCP for Low La...
by _zoltan_ on Wednesday August 28, 2024 @02:40AM
(#41376637)
With IB (Quantum X800) you have 72x physical OSFP ports that can do 144 ports. Each electrical lane runs at 200G-PAM4, so it's 144x(4x200G-PAM4).
With the SN5600 for Ethernet (Spectrum-X), which is 64 physical ports, you're running each port at 8x100G-PAM4).
There's also other differences, such as port counts. AFAICT Spectrum switches at 400Gbps have up to 128 ports whereas equivalent Infiniband NDR Quantum only have 64 [0].
When building clusters of 32K+ GPUs the network cost, power, transceivers etc start to add up.
[0] https://www.semianalysis.com/p/100000-h100-clusters-power-ne...