1. Field of view is limited to existing optics miniaturization
2. Subtractive shading (rendering black) might not be solvable
3. Variable focus objects in the same scene requires projecting n>2 significantly different wavefronts - not solved how to do this with a single vibrating element
For example, close one eye and hold the tip of a pen about an inch in front the other and you'll see that it doesn't actually block any of the world.
The blocking all light approach also allows you to hide other potential weaknesses of a device. For example, a lower field of view is much more distracting in a pass-through AR device as you still have your full peripheral vision. VR devices will generally black out the light outside of the FOV making it easier to ignore.
I mean... theoretically you can if you can match photon for photon with the inverse phase. Difficult, I know :)
Moore's law works great for semiconductors, but Maxwell doesn't negotiate ;)