kguttag
The camera should take pictures with the IBIS fractional picture shift and save the RAW files. They should give the option of how many "cycles" (go through all the shift orientations more than once) of pictures to take. With that level of information, smart software will be able to figure out and deal with at least small hand motions and considerable motion in the subject.
Smartphones are already using computational photography, combining multiple photos for things like panning and taking pictures in the dark. For a dedicated camera like the R5 mk ii, I would want the camera to save RAW images that can be put together later by "smart" software on a computer with greater processing and memory resources.
I was a bit sloppy in that regards. I was more worried about what was going on. But when combined with the word "passthrough" to see the real world, I tend to drop back/slip to the term AR passthrough which is what this type of thing was called for years. MR, as I remember it, is a more recent term used to distinguish different types of AR. Then things flipped and XR (=AR/VR/MR) was used to mean was AR used to mean.
My guess is that late in the program, the importance of passthrough was elevated, perhaps in response to Apple rumors, but they were stuck with the hardware that was in the pipeline. The passthrough available is a big improvement over the Quest 2 for use in seeing the grouse-level surroundings, but not up to regular use in full Mixed Reality.
There are probably decades of "optimizations" left to do.
But it be ridiculous to thing that people are going to be going out in public with a VR headset with cameras (known as Passthrough AR). On top of a number of human factor issues, it would dangerously block a person's view of the real world.
I was impressed by the size of the Avegant engine and the transparency and lack of forward projection by the Dispelix prototype. They are claiming they will get 2,000 nits to the eye out of the design which should be good enough for outdoor use IF they have some form of clip-on sunglasses (2,000 nits is not enough for outdoors in full sunlight without some help).
The current Avegant engine has an optical component in that was depolarizing the light from the LCOS display and losing contrast so they wanted me to wait to take through the lens pictures. The image from the current prototype looked sharp but did lack contrast.
That said, I'm very skeptical that a VR headset will work in place of a computer monitor for long term use with business computing.
The problem with medicine is that is it a tiny market. You can't justify the type of money they were trying to raise for that market. Under Abovitz, they were always a swing for the fences company.
All the above said, nobody I know of is making money selling AR headsets today. They are either subsidized by VC or other investment money or by big companies funding R&D efforts. The market is not other product areas where you can make money with an MVP and grow the product.
You might want to see this video (queued to his discussion of Kura at CES 2022: https://youtu.be/S0heZVN5NCs?t=1328
I am seeing a lot of progress in some areas and will be publishing an article later this week on them. In particular both Dispelix and Digilens have made considerable progress on the "glowing eyes" issues (Dispelix all but eliminates it). Avegant has a very nice small LCOS light engine that pairs nicely with the Dispelix waveguide.
I think the biggest problem for AR is that the expectations are very high and the physics is very tough. Many physical optical features within a few wavelengths of light were diffraction ruins everything.
I think a large part of this is that they were building the ML2 for the high end consumer video game market when one day they they were told it was too expensive and it would now be an enterprise product. I hate the display quality of the HL2, but at least they improved ergonomic and interface issues over the HL1, whereas the ML2 repeats the mistakes of the ML1.
The rumor mill is that Microsoft has slowed down development of HL3 development with many people leaving. Most famously, Bernard Kress, and optical architect of Hololens 1 and 2 has left for Google.
There were very few Magic Leap Ones ever sold. The number I hear is between 5,000 and 10,000 units sold for money with about as many units given away.
Hololens 2 seems to be shipping about 70K to 100K units per year but it is not clear if that has peaked or not. The display quality is very poor as I have written about extensively, but the ergonomics are good enough for some industrial applications.
I get the sense that ML2 was too far gone and with money running out to react to ML2 and thus still has terrible ergonomics. With blocking 80% of the light, not supporting ordinary glasses, and having a cable which is a snag hazard, the future looks bleak for Magic Leap 2 as it was for ML1. I don't know if they would have been a success, but at least they would have had a shot in the same space as HL2 if they had reworked the serious ergonomic problems. The dimming feature is a differentiation that resulted in them missing the market altogether.
KGOnTech: Meta [Facebook] Buys Imagine Optix
Doing a little math, that is only about 13 pixels per degree or 4.5 arcminute/pixel. Typical VR headsets are about 30 pixels/degree (2-arcminutes per pixel. For optical text reading like with a computer monitor, I consider 40 pixels/degree (1.5arcminutes/pixel) the bare minimum.
There is also some scintillation from the beaded screen. Not terrible, but enough that it would not be good for text. So even if the resolution was higher (which is possible with this technology), I'm still not sure it would be good as a computer monitor.
The case where it works best seems to be in "tabletop" applications, and it behaves best when the viewer is about 45 degrees to the surface. Business and military applications for things such as "sand tables" also fit this model.
I'm not a lawyer, but I think it was done this way so the I.P. would immediately become J.P. Morgan Chase's in the case of default. We don't have access to the contract between Magic Leap and JP Morgan Chase to know what the exact terms and condition.
Regardless, whereas before Magic Leap was able to raise money without putting up the I.P. as security for a loan.
What makes it "news" is that Magic Leap was able to raise $2.6 Billion without having to resort to using the I.P. as collateral. It usually does not make the previous investors happy and you usually resort to this when you can't raise money any other way. It is also a one-trick-pony in that you can't do it again and future potential investors know that the I.P. is already gone if the company goes bust.
They just announced a $280M investment from NTT Docomo, but base on my and other people's estimates, Magic Leap was running out of money while burning through about $600M/year so that only bought them about 5 months of runway.
I have up the capacity, but when the cite hits #1 on Hacker News (as it is right now), it looks like the traffic is still overwhelming. It probably does not help that I put the larger associated document for people to download (I was going to put a link up but the link was not working in the Chrome browser).
KGOnTech: Magic Leap One – The Terrible View Through Diffraction Gratings
There are so many hits on the site that it got throttled.
I just upgrade the hosting plan to hopefully solve the problems, but I won't take effect until 2AM PDT.
Basically you have multiple fibers going in a circle each with their center of origin it becomes impossible to get them to act like a single image for use in near eye optic.
What the layman would not understand is that this is very different from image stitching on a projection screen that in part relies on the light being diffused/randomized by the screen. In the case of near eye optics, there is no way to get the multiple projector image collimated AND seamlessly put together.
It is yet another example of a good con has to seem believable.
The parallel fiber idea is "optically silly" but it takes some understanding of light to prove it is impossible too. It is yet another example of trying to fool people. I did try and explain this before on my blog over a year ago. See http://www.kguttag.com/2016/11/20/magic-leap-separating-magi... and scroll down to the Appendix at the bottom.
Basically you have multiple fibers going in a circle each with their center of origin it becomes impossible to get them to act like a single image for use in near eye optic.
What the layman would not understand is that this is very different from image stitching on a projection screen that in part relies on the light being diffused/randomized by the screen. In the case of near eye optics, there is no way to get the multiple projector image collimated AND seamlessly put together.
It is yet another example of a good con has to seem believable.
I can't say who I am going to see as a lot of it is private. I will be spending a LOT of time in the Main Convention center South Hall with the AR and VR companies. The special section for AR and VR has moved from the back of South Hall to near the front this year. Then you have to look for the more established companies that booths elsewhere.
The CES tools for finding booths and vendors have improved dramatically form where they were a few years back. You can search based on what you are trying to find. They also have a smartphone ap that comes in handy at the show.
The Fiber Scanning Display is in a whole different category. This looks like pure lying on Magic Leap's part.
In this particular case I have tried to lay out the math that proves that Fiber Scanning Displays are never going to support high resolution. No amount of money or time is going to make Fiber Scanning Displays work. I picked this case because the math was simple to follow and easy to prove that it was not possible. Yet it has been a key point in Magic Leaps Presentations and patent applications through to the latest ones printed.
I think it is a very valid point to asked how a company could have been presenting something that was not possible, why people invested in a company saying things that were not possible, and why Magic Leap is holding to the fiction that it might be possible.
It seems to me to be a pretty simple question. Were they lying or did they not know what they are doing?
There are other areas of optics and image quality where you can argue about whether the image quality will be "good enough" but Fiber Scanning Displays is a black and white issue.
People today are all the time waving their hands and invoking "Moore's Law" and saying "Steve Job did this" and applying it to everything (I see it all the time in AR and Optics) and therefore anything is possible.
Similarly it looks like Avegant is calling what they are doing "Light Fields" because Magic Leap was calling Focus/Focal Planes "Light Fields".
Loosely conceptually, Light Field work in the horizontal direction relative to the light rays from the image source, where Focus Planes work in the vertical direction.
I have not done much with Light Fields personally, but Nvidia has published some good work about them (https://research.nvidia.com/sites/default/files/publications...) using what I would call "classic light fields." Gordon Wetzstein has presented how to do light fields with much fewer sub-elements (https://www.youtube.com/watch?v=itSjM_OzVtA) and there is a lot of other material out there on "compressive light field displays" that he has been working on for a long time. He claims to get decent results with just 2 fields. BTW, I highly recommend reading about everything I have seen from Dr. Wetzstein, he does an excellent job of explaining complex optics concepts.