I had an idea to do something similar once for a Uni dissertation, involving a rift mounted with two cameras to do a very hacky and cheap prototype version of what you've described.

My supervisor shot it down because "Google glass will do that" :(