I wonder if it would be possible to move some processing into the display. It's not the same, but in particle physics pixel detectors, which are basically CCD sensors, people have been working to move the readout electronics into the pixels. There, the motivation is radiation hardness and cheaper production. Here, it could be decreased latency. What if you could put a super small shader in each pixel, and do a final processing step like 'timewarp' right there? Or a part of your graphics memory is directly your display, no bus in between?

That being said, I tried a couple of VR solutions and I don't really have a problem with latency. What I find more irritating is that I cannot focus well on different planes. You do have the effect when you cross your eyes that you put images from different planes together - although badly, because even if you don't see the pixels, you do get some kind of moire pattern when shifting planes (I can't describe it better). And what you don't have at all is depth blur. I think both can only be solved by true light field displays (if that is even possible). I hope there will be a breakthrough with holographic projection at some time.