What you're describing is already a thing, UE4 just shipped an experimental implementation in 4.15.

Search for Monoscopic Far Field Rendering here: https://www.unrealengine.com/blog/unreal-engine-4-15-release...