Re:Neural Supersampling for Real-Time Rendering
by vlovich123 on Saturday July 04, 2020 @09:18PM
(#23736014)
This is an oft-repeated & poorly sourced claim. My view of the landscape is that the fundamental shift that has happened in VR with ATW hasn't been fully appreciated by those working outside of VR.
ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable.
Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms.
Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver.
This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.
ATW means that your head rotational latency is fixed at your display rate. Now the content still has a higher motion to photon delay. The user's perception of motion sickness though is not about that though. It appears to be primarily incongruity between where your eyes are focusing & the signal from your inner ear. ATW is relatively cheap & well understood so high refresh rates with a deep pipeline aren't totally unreasonable.
Once you have that, the total end-to-end latency matters in a different way. Now you're focused on the content not feeling laggy. That's where prediction comes in. 100ms of end-to-end latency is probably outside the realm of what is doable today. With algorithmic improvements, increases in computational power, improvements in body modelling & neural networks, it's not unreasonable to start thinking about 100ms.
Anyway, the mental picture I meant to paint you was what if you thought about all the content in terms of how mobile GPUs (& desktop GPUs under the hood) think about it, in terms of tiles. What if your physics simulation only bothered to simulate the current tile's worth of content before handing off to the rendering stage. What if rendering focused on that tile before moving onto the next? Then you submit a tile's worth of content to the display. Then your display isn't refreshing the entire panel uniformly but instead refreshing each tile offset from the other. On top of that you bake time warp directly into the display driver.
This is a monumental undertaking. The reason we haven't historically done that as an industry is because HW speeds with the existing architecture have increased far faster than the SW architecture could mature to take advantage, so from a cost perspective both the HW & SW vendors would be idiots to try to do this. However, HW speeds are petering out & even purpose-built accelerators can only do so much. At some point the architectural shift will happen.