> the network topology and the network switch itself also can make a huge difference depending on traffic conditions; so you might have tons of fat NICs per GPU but if all of them want to alltoall, you better have a ton of cross section bandwidth.
This is the beauty of the new nvswitch chips and the infiniband networks instead of ethernet. Anyone who is doing this is setting up a fully switched high bandwidth infiniband network with ridiculous traffic between them. Nvidia purchased Mellanox a year or two ago- combine that with the ridiculously awesome nvswitch in the A100 dgx machines and there's a huge jump in cross chip traffic ability. At the same time though a decent mellanox router is probably going to set you back $30k.
I'm not aware of any cost-effective switch that permits scaling all v all to arbitrary sizes. THat's my point. modern infiniband uses nearly all the same tech as previous supercomputers, but with faster interfaces, and more of them. For example the facebook cluster is a dual-layer clos network, which is one of a few cost-effective ways to get very high randomly distributed traffic, but all v all communication scales as n*2 and n*2 wires gets expensive fast.
Better to find algorithms that need less communication, than to make faster computers that allow you to write algorithms that needs lots of communication. Otherwise you'll always pay $$$ to reach peak GPU performance.
I always wonder about performance on these clusters. Back in MY day, I'd wait a week or more for results from my jobs, and immediately resubmit to wait two weeks in a queue for another week of runtime and do lots of data processing in the downtime. Then, I moved to cloud and decided on "what can I afford to do overnight" (IE, I set my time to result to be about 12 hours). I have a hard time justifying additional hardware to get results in 10 minutes versus a day, it seems like at that point you're just using it to get fast cycle times on new ideas, but who has new ideas every 10 minutes?