It won't be 5x slower, it would be 20-50x slower if you would implement it as you said.

You can't just "add more ram" to GPUs and have them work the same way. Memory access is completely different than on CPUs.