That's true of any system. A "real" 4GB that touches the 4.1GB region will be forced to use CPU main memory over the PCIe bus.

So the 0.5GB region is still faster than PCIe IIRC. Not ideal of course, but still better than a bunch of PCIe transfers.