> you have to throw out basically everything in GPU memory and reset it all

This is not an inherent limitation of CPUs but a part of Windows' exclusive fullscreen concept. Just another thing that was simply accepted as the way things are instead of being improved (until exclusive fullscreen went out of style).