At some point your GPU runs out of things to do.
The reality is that greater horsepower allows for greater abstraction, and easier to program APIs. Increasing developer productivity 2x reduces performance 10-100x, or something like that. So there’s never “enough” performance for the same reason there’s never “enough” powerful/usable APIs.
Audio used to be really difficult to process in real-time but now it's trivial. There's only so much audio processing you can do before it's ridiculous and pointless.
The same goes for video. Once you have, say, a 40K display for each eye at 244Hz there's no point in going for more pixels or faster refresh rates. If a GPU can handle that, easily, then that GPU will probably be best put to use doing other things in addition to rendering graphics.
Memory is not tied to your senses, we can always find uses for more. Audio and video are, and at some point it's as good as real.
High res stereo at 120Hz is never going to have to same graphics as the latest high budget big game release. Current GPUs are already very powerful, but if people have the expectations of getting the same graphics when they use VR they are going to be very frustrated.