Magic Leap should skip the fancy stuff (mixing virtual scenes with real), at least at first, and focus the many other useful features of a great head-mounted display system - think mobile notifications, video calls, web browser, etc.
It could easily replace smart watches and later cell phones and computer monitors without solving the latency issue.
Edit: you'd still have the motion-sickness challenge, but perhaps at least the 'layers', so-to-speak, wouldn't appear separately.
zipppy says "what if AR was delayed 15ms to make it equiv. to VR".
Zipppy's intent was to make AR compete with VR, not HVS.
Because we have the head tracker recent history, we use prediction on pose trajectory, and can effectively know where the head pose will be at the time the current rendered frame will be displayed. And use that predicted pose to render the scene. That type of optimization won't be possible with see-through VR or AR.
The second optimization is timewarp, where the rendered scene is distorted in screen space after the fact, based on post-render tracker data (just a few ms before display). I wonder if that type of optimization would create artifacts in AR.
1 - the robot moving behind the table leg (ie you have to do depth recognition of objects in the scene)
2 - the user's hand interacting with the artificial elements in the scene. Some code had to recognize a hand and figure out which element it was touching.
What strikes you as the hard parts of those videos besides the real-time requirement?
I remember reading a while ago about how smart tvs were using ANNs for upscaling, so it has been done at scale. rimshot
(2) Upscaling ANNs process rather small image neighborhood radius, and required processing power is on the order of O(r² * log r), and if a minimally recognizable cat is 50x50 px and for upscale you use a very large window of 16x16, that's 14 times already.
16x16 is a very small window, I have no idea what they're using for TVs, but 128 isn't uncommon in post production ANN upscaling. Also consider the fact that ANNs have not received anywhere close to the level of attention in optimization that compilers have, so there is also a lot of potential slack to be taken up if real time processing demands it.
2 bullshit CGI "this is how we hope it would look like if it was real" demo
Few months ago their apparatus was one color only, stationary and the size of a desk. Now all of a sudden can be strapped to a camera and does colors? color me sceptical :(
My personal take on this is that they may indeed have some very good, if not revolutionary, display technology. However: The big, big obstacle to delivering credible AR is latency. Contrary to VR, true see-through AR needs to have total latencies (device motion --> display photon hits the retina) of no larger than 10 - 15 ms max. The reason is that in see-through AR you're essentially competing against the human visual system in latency and the HVS is very fast.
Moreover the HVS is also extremely good in separating visual content into "layers". Whenever two things in your field of view don't move in perfect continuity with their surroundings (as it is when there AR content overlaid with latency) your brain will immediately separate them from one another, creating the impression of layers, and, in the case of see-through AR, breaking the AR illusion.
So right now I'm a semi-believer. Iff they can sort out the latency problem and deliver stable yet ultrafast tracking in a wide variety of conditions (also by far not a trivial problem) then this has a bright future.