The update rate has to be very high and the alignment perfect or it will drive users nuts. Which means it will probably be done soon, although not in this round. Doing it with lightweight short path optics is also difficult. All this stuff with diffraction gratings and Fresnel lenses makes for a crappy image. That can sometimes be overcome; the Polaroid SX-70 camera had a very compressed optical system with Fresnel lenses without compromising image quality too much.[1]

The other option is simply to show the real world via a camera feeding into the system. That's been tried many times, and it works OK, although lag will make users faceplant or ill.

[1] https://www.technologizer.com/2011/06/08/polaroid/sx70-light...