I'm not aware of any cost-effective switch that permits scaling all v all to arbitrary sizes. THat's my point. modern infiniband uses nearly all the same tech as previous supercomputers, but with faster interfaces, and more of them. For example the facebook cluster is a dual-layer clos network, which is one of a few cost-effective ways to get very high randomly distributed traffic, but all v all communication scales as n*2 and n*2 wires gets expensive fast.

Better to find algorithms that need less communication, than to make faster computers that allow you to write algorithms that needs lots of communication. Otherwise you'll always pay $$$ to reach peak GPU performance.