http://uploadvr.com/magic-leap-how-it-works/
Magic leap is a pair of glasses you wear on your face that displays AR imagery through a clear glass lens. There's a cable leading from these glasses to your pocket where you have a tablet-size device doing all the heavy lifting, most likely using mobile SoC's. Pretty sure you can't run a desktop CPU and GPU on a pocketable device.
You don't get 1080gtx performance or anything remotely close to that with a setup like this. Its mobile-level graphics. Oculus's recent showing of its mobile SoC-based Santa Cruz prototype shows fairly poor graphics compared to PC-based VR like the Rift or Vive. I imagine the Magic Leap has less pixels to push considering it isn't doing any backgrounds, but still, those very impressive marketing shots of high resolution dragons flying around and interacting with you might not really be possible without a lot of compromises on graphic quality, fov, framerate, etc. Worse, no one wants to talk about transparency, especially in well-lit rooms. The few hands-on Hololens reports we've gotten make this out to be a big issue, as well as fov, and fundamentally the Magic Leap and what hololens is using could be the same or very similar technologies.
My gut feeling is that no one has yet to handle the transparency issue (and this is why MS can show off the hololens with confidence) and because of that it makes sense for Magic Leap to keep its product secretive. The tech press was not too kind to Hololens prototypes due to transparecy issues and Magic Leap doesn't need that kind of negative press as it continues to fund-raise and develop its product.
> You don't get 1080gtx performance or anything remotely close to that with a setup like this. Its mobile-level graphics.
A.) the pocket device is the light source array that feed the scanning fibers. A processor drives each light source linearly. Neither the light source nor the driving processor are likely to be head mounted for quite some time, but I don't see that being an issue given the dramatic improvement over existing interfaces (i.e. I'd rather have a pair of glasses with a tether than a helmet any day of the week);
B.) In light field VRDs, resolution is not limited by active matrix array addressing as it is to a degree in LCD displays of mobile devices. Instead, resolution primarily depends on how quickly you can resonate the tip of a scanning fiber and concurrently how quickly you can pulse that fiber's light source in a synchronized manner (each pulse matching the bit color/intensity of the intended pixel);
C.) Multiple fibers form a composite image;
D.) Maximum resolution is only needed in the foveal region (i.e. a few deg about the center of user's eye gaze at any given time), so eye tracking can be utilized to minimize processing. The remainder of displayed content is low resolution, albeit still critical, peripheral imagery.
My team & I have been batting around with a slightly different approach to the light field VRD, which we're thinking about simply releasing as a FOSH project.
More details would be fantastic
The resultant display has an inherently smaller exit pupil, but we'll layout a method to actively expand the exit pupil, enough to support typical maximum eye rotation range and minimum eye pupil dilation (~2mm) without vignetting the virtual image with natural eye movements. The benefit of this method is virtually unlimitted FOV.
Certainly still in development, but we think releasing what we've come up with thus far into the wild as FOSH and seeing what the crowd can bring to the design is a fun idea.
I'm currently working on a simple "Hello World" write up that will explain the device components, some basic plans for prototyping done to date and the development hurdles that still need to be overcome. I'm planning to use this to introduce the project. It's definitely not an easy project, but I don't see any component that can't be fabricated by a crafty homebrew engineer.
The Oculus et al need to render a rectangular matrix of evenly spaced pixels. If you want a larger field of view or higher resolution this comes with an exponential increase in number of pixels. With a high frequency scanning display and eye tracking you are not locked to a static spatial resolution (see https://en.m.wikipedia.org/wiki/Foveated_imaging).
There's a whole bunch of other problems when it comes to a render pipeline for something links this, but at least on the physical front it enables super high perceived resolution with less overall points of light to render.
It was interesting to hear Carmack last year talk about re-introducing interlacing as a way to reduce latency.
Although, I'm just reinforcing what you're saying =)
For VR, the consensus I've seen seem to push for multi-sample AA > full-screen AA > temporal AA (I feel like I've even seen no AA is better than temporal AA). I'm not quite sure if it's performance, architecture (they really prefer forward renderers instead of deferred renderers for low latency), or aesthetics. In the little bit I've done, when just playing with knobs temporal AA looks better to me. Without AA specular highlights are way too distracting and pop too much.
This reference[2] talks about the extra velocity buffer needed for temporal AA and the fact it tends to over-blur which can fuzz out fine details (low resolution is a touchy subject in VR).
There's a lot of moving pieces when making decisions. In talking about interlacing, he was ideally talking about not having to create a whole image buffer before sending data to the hardware--just rendering parts (well, scanline) of the image that moved.
[1] https://youtu.be/gn8m5d74fk8?t=10m54s [2] https://www.youtube.com/watch?v=Xk3WUk5T2TQ&feature=youtu.be...
'Enough' is retina resolution, but only on the fovea, and way less the further away you go from the fovea. That can significantly decrease the amount of work needed to render a scene.
Moreover, color-wise, you don't need color except for the center of your field of view (you need cones to see color, and cones can only be found in the fovea).
Building such a screen probably isn't that hard; you would need to build new production lines because everything out there is built for rectangular grids of equal-sized pixels, but that probably is 'just' a matter of investing money and time.
Writing the software to efficiently draw scenes on such a screen probably is a bit harder; there will be lots of optimizations that need to be discovered (what's the equivalent of Bresenham line drawing, for example?)
The biggest challenge will be to keep the high-resolution part of your screen centered on where the retina looks. That requires eye tracking and some way to move your screen. Saccades move at hundreds of degrees per second; you would have to match that.
If that's what they are building, and they succeed, I think it will take over the world, even if they initially sell it at $5000 an eye.
How does that work? Shouldn't it be just linear in FOV? Or quadratic if you are talking about solid angle? This doesn't seem like one of those things that grows exponentially.
- FOV is apparently their big advantage, along with focus zones
- They could do hard occlusion with an LCD mask. Or they could punt til v2, and soft-occlude with blur using their stacked diffraction plates.
Unfortunately, you cannot easily achieve "hard occlusion" with an LCD mask. Pinlight display is a project that did explore that idea: http://pinlights.info
LCDs can attenuate individual rays of light. However, the images we see of real objects in our environment are generated from light in the form of planar wave fronts. So, if you have a near eye LCD micro display and you turn on a single LCD pixel in an attempt to block light emanating from a certain point on a real object, most of the light emanating from that point will simply go around the LCD pixel, enter the pupil and for a point on you retina. The total brightness of the image will dim a little dependant on the size of the LCD pixel, but not by much for a single microdisplay pixel (i.e. a few micron pitch).
The Pinlight papers do talk about ways around that issue, using LCD diffraction pattern masks, but I still think the residual diffraction and reduced overall lens transparency associated with the LCD micro display makes the approach an even bigger challenge.
Could you emulate that with a transparent LCD display? Yes, but that display would need to be large and about 1.5ft from your eye. Not really practical for a near eye (head mounted display).
If you try to accomplish this occlusion directly with an LCD microdiplay (such as those found in many projectors) in a near eye application, at best you end up with a blackish smudge with very fuzzy edges. To see this in action, look at Figure 11.D in the following paper:
http://www.cs.unc.edu/~maimone/media/pinlights_siggraph_2014...
>[0216] Such may be used to cancel light from the planar waveguides with respect to light from the background or real world, in some respects similar to noise canceling headphones.
http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=H...
> http://appft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=H....
I believe this particular excerpt is referring to the use of the layered and controllable DOEs to curve light projected from their scanning fibers that ultimately forms the virtual object image on the retina, rather than using the DOEs to attenuate ambient light (see my reply to Kelsolaar above).
However, if I'm interpreting their literature correctly, you are correct in that they are able to use (at least a portion of) these DOE layers to generate the moire patterns that will ultimately be capable of form black/clear image masks.
On the other hand, I believe ML is proposing to use a gratings to display black/clear images. In this setup, the incident ambient light wavefronts are selectively redirected using bragg gratings (why they keep saying Braff gratings" in their literature is beyond me). This technique has been around for many years and is still captivating to watch even in this day and age - see:
https://www.google.com/url?sa=t&source=web&rct=j&url=https:/...
The realization that this affect retains its "sharpness" for near eye applications in concert with (I believe) the realization that you can utilize the electonically-controlled diffraction grating layers you already have in your device (for the purpose of controlling wavefront curvature to support eye depth accommodation) depth is really quite clever. It's truly an exciting prospectct!
One of the researchers posted a 30ish min video on the hardware inside the Hololens on YouTube. It's this HPU / sensors that Microsoft is licensing to the hardware manufacturers for the upcoming VR and AR headsets. We'll know more December 8th, and after their hardware expo later this year.
If we want current gen graphics we're more than likely looking at a PC backpack solution. More than likely we'll probably be looking at some low watt solution, like the 35W AMD chipset in the Solus Q and accept a lower level of graphic quality, but still well above mobile standards. This may involve wearing a battery belt or battery hip-pouch for extended use. For lower run times the battery can be part of the headstrap with no need for anything to be pocketable.
Ugh. I just imagined a room full of people sitting around a conference table with VR headsets on. Not a pretty thought indeed.
But their later patents showed whole arrays of these fiber projectors. They are probably limited by the optics in-between the fiber and the eye and by the display controller bandwidth more than anything.
But might was well stay hopeful for as long as possible.
Problem is that would require at least two huge leaps in technology one in display technology and one in compact computing power, probably more.
Also considering how litigious/secretive things are — it must suck to work on something for so long and be contractually obligated to avoid talking about it in public.
It would certainly be a much more complex solution that would still have inferior results compared to computing everything locally (just like cloud streaming now). Just that it's not "change the laws of physics" impossible to get to a workable implementation.
See also John Carmack's comments from 2013 on latency mitigation: https://web.archive.org/web/20140719053303/http://www.altdev...
Cloud gaming has not done all that well; it's right at the edge of feasibility due to latency. VR moves it back off the edge to "unquestionably infeasible". If VR gaming were to take off and become a dominant paradigm it would make cloud gaming even harder to justify.
I was completely skeptical at their claims with respect to latency. I never believed they could get the service to work, and for enough people to make it profitable. It appears quite a few people were able to enjoy the service at decent enough framerates. Which is surprising.
A few weeks ago my cousin ran games at 1080p on an EC2 GPU instance and used Steam's In-Home Streaming to play remotely and it worked perfectly fine.
I think the latency is now there, if you have a fast connection and live relatively close to the datacenter.
Pseudo-real-time gaming is possible, but only for regional servers. Live global gaming is off the table permanently unless known physics changes; speed of light delays are larger than human reaction times. Right now, the best we can do is predictive techniques and continental servers, which have barely enabled MMO shooters.
VR gaming worsens the problem - 80ms delays in VR are nauseating and game-ruining. Global latency isn't a solvable problem via money or devotion.
There is a limited useful spectrum that's shared between everyone in the area. You can't just throw more bandwidth at the problem.
(It's not actually required, but the efficiency loss between H.264 and HEVC for 4K is pretty massive.)
"Throw out your PC, your laptop and your mobile phone, because the computing power you need will be in your glasses, and they can make a display appear anywhere, at any size you like."
Yeah sure, they not only made a new display tech but integrated the computing power of desktops or laptops into glasses. I wonder how they cool the GTX 1080 cards on your head...