It doesn't matter if it bypasses pcie. The fastest available ethernet interface today is 100Gbps. Pcie gen3 x16 will get you to about 115Gbps, so the limiter is the interface speed. Pcie4 should be common when 200G interfaces come out. This is why an external nvswitch can make network switches vulnerable in this market.

Power 9 has pcie4, so you could theoretically already double your pcie bandwidth with a mellanox card that supports it, but from my previous point, it doesn't matter. Those nvlink numbers are disingenuous. That's the total links to/from a GPU. All available power 9 machines split those links between the other GPUs and CPUs, so you really only have up to 3 of those in the best case, or 75GBps. That's only a bit more than double pcie 4 at that point. And also, by using those nvlinks back to the processor, you are making a decision that you don't need as much bandwidth between GPUs. With the nvswitch architecture, you get the best of both worlds with a full, non-blocking switch.

There’s actually six 50 GB/s nvlinks on each Power9 cpu (300GB/s), 2 or 3 to each GPU, depending on if you have a 4 or 6 GPU system. I’ve seen speculation that the DGX-2 nvswitches has enough free ports to fully connect in 2 Power9 cpus with 300 GB/s each, that would be an interesting system..

This is true, but it is essentially the same situation as using the CPU for processing.