Data has to be copied up to the GPU for processing. It has to be brought down to be sent as a result set somewhere. If everything you're doing fits in GPU memory, you've only done that once. If it doesn't, you need to copy data up and down. You may need to do that for intermediate results too.

This adds up.

Power 9 has pcie4, so you could theoretically already double your pcie bandwidth with a mellanox card that supports it, but from my previous point, it doesn't matter. Those nvlink numbers are disingenuous. That's the total links to/from a GPU. All available power 9 machines split those links between the other GPUs and CPUs, so you really only have up to 3 of those in the best case, or 75GBps. That's only a bit more than double pcie 4 at that point. And also, by using those nvlinks back to the processor, you are making a decision that you don't need as much bandwidth between GPUs. With the nvswitch architecture, you get the best of both worlds with a full, non-blocking switch.