While reading about OpenGL programming, one of the first things nearly all books mention, is the cost of moving data from your system to the GPU's memory.... So I assumed that's still true, is it not? Adding more GPU's does not change things as much as reduce the problem by providing more space.

Memory bandwidth is usually the bottleneck before PCIe.

You see this effect in other things too, such as Intel NICs being able to place packets directly in L3 cache because avoiding main memory is such a huge win (for both bandwidth and latency reasons):

https://www.intel.com/content/www/us/en/io/data-direct-i-o-t...

That is true in a sense of: Of all the things in the graphics pipeline moving data between CPU and GPU is the slowest thing you can do. But that doesn't really mean the PCIe bus is slow. The truth is that most machines have a single GPU, connected to the CPU with all 16 PCIe lanes but they usually hit their performance cap utilizing much less then that.

This is coming out to PCs nowish: https://devblogs.microsoft.com/directx/directstorage-is-comi...

Nvidia's 3000 series will support it, the new consoles have it one form or another, and we can safely presume AMD's 6000 series will support it as well.

Under Windows, obviously. Linux will be a completely different bucket of chum.