For one, the positioning of the display and camera. The display is elevated so you don't see it directly in front of you. You have to actively look at it. The camera isn't centered in your face so it doesn't line up with your vision.
I agree the display is only part of your vision. That's suboptimal but I'm not sure it's a showstopper.
The camera not being centered on your face is not insurmountable. (In fact, you'd want it to be centered on your right eye, not your face.) The project I linked to above aligns the camera with your vision. It doesn't currently work for different depths of field, but I'm working on that - this relies on your image recognition being able to determine the depth of the recognized object, e.g. by knowing its physical size.
(I'm playing with it as an augmented reality device and have written a calibration library to get the display aligned with the camera - see https://github.com/matt-williams/Optometrist. I've been working on hooking this into OpenCV which I'm hoping to upload in the next few days.)
One valid answer to "why not" might be power/heat - if you run it for a long period of time, your battery dies and your hair sets on fire. ;)