I also really don't know why he decided to deemphasize the perspective and depth correctness so much. He mentions it here:
> In this case, they were willing to sacrifice image quality to try to make the position of things in the real world agree with where virtual objects appear. To some degree, they have accomplished this goal. But the image quality and level of distortion, particularly of “close things,” which includes the user’s hands, is so bad that it seems like a pyrrhic victory.
I don't think this is even close to capturing how important depth and perspective correct passthrough is.
Reprojecting the passthrough image onto a 3D representation of the world mesh to reconstruct a perspective-correct view is the difference between a novelty that quickly gives people headaches and something that people can actually wear and look through for an extended period of time.
Varjo, as a counterexample, uses incredibly high-resolution cameras for their passthrough. The image quality is excellent, text is readable, contrast is good, etc. However, they make no effort to reproject their passthrough in terms of depth reconstruction. The result is a passthrough image that is very sharp, but is instantly, painfully, nauseatingly uncomfortable when walking around or looking at closeup objects alongside a distant background.
The importance of depth-correct passthrough reprojection (essentially, spacewarp using the depth info of the scene reconstruction mesh) absolutely cannot be understated and is a make or break for general adoption of any MR device. Karl is doing the industry a disservice with this article.
I also can't tell if KGuttag believes that you can simply pass the video feed through to the user and not have an absolute usability disaster. His paragraph about the "pyrrhic victory" sort of implies that not preferring depth-correct scene reprojection was an option. In my experience, that is absolutely non-negotiable in terms of shipping a device that's usable to the majority of consumers.
Another reason why folks also don't understand why passthrough is so energy intensive. You're also running an entire additional pass of steresocopic spacewarp on what is essentially four different camera feeds, on an incredibly high resolution scene reconstruction/SLAM mesh.
Would it be nice to be able to actually read text and not have distortion when objects get too close? Yes, of course. But it's not necessary.
Which unfortunately includes your hands.
But it is a big turn off to have no visual idea of what is going on around you. Just to be able to press a button and be able to see that there is someone standing around you, or you are too close to a couch, is a big win for a VR headset in my opinion. Obviously the visual fidelity will improve in the display and the pass through. But just starting to get some pass through is very nice.
Until AR display tech gets significantly better I think AR based on passthrough cameras is the only sensible approach.
Would you mind giving user-facing examples to illustrate the difference? As in something you consider AR but not MR and vice-versa.
I've googled a bit but I don't understand the distinction between "enable display and interaction with the real world but with [virtual] augmentations" and "bring virtual objects into the real world and allow you to interact with them". As someone with little AR/VR/MR experience these sound like two ways of phrasing the same thing. Thanks!
An AR example would be the classic overlay of text on a sign you are looking at with a version that is translated into your language.
[0] https://www.oculus.com/deeplink/?action=view&path=app/158709...
Where would you set Pokemon Go’s “AR view” [0] kind of implementations for instance ? (First link I found was for a headset, but I’m also actually curious how you qualify the phone only version of it)
Or what would it change if in you Gravity Sketch example each user would be seeing colors or rendering adapted to their vision/preferences making it a separate experience for each of them ?
This image show an example of it: https://www.appliedart.com/assets/images/blog/_newsFeatured/...
I hope the folks there can figure out the operational difficulties, and find a way past whatever org chart drama is the source of the dysfunction. The Quest 2 has been a delight, but enough time has passed that the limitations are starting to chafe. I would love to see a true Quest 3 made my people who actually like the things.
[0] https://medium.com/@jas0njames/the-primacy-of-shipping-94b8e...
I can understand they try to see far to anticipate a future where they'd make a lot of money being useful, but there's such low latency and richness of perception with normal atomic real world interactions that I'm not sure we can or want to replace it with something that simply adds a paid intermediary.
It s like adding a blockchain to transact cash for instance.
Moreover, "Mixed Reality" strikes me as marketing-speak. Compare the search trends for AR [1] vs MR [2]. While both terms have been in use for a long time, AR was always more prevalent, with a notable rise in interest around 2009. My first encounter with the term MR was in marketing materials for Microsoft's HoloLens circa 2015. Now compare the search trends for MR [2] with HoloLens [3]. Searches for MR don't begin to trend upward until after the introduction of HoloLens, where Microsoft's marketing materials tried to make a big deal of said distinction and the rest of the world started to run with it. As a supporting argument for the original broader usage of the term AR, I'll cite discussions of a 2007 anime series, Dennō Coil [4], which explored various examples of fantastical AR usage. Probably just a coincidence, but the search trends for Dennō Coil [4] do precede the search trends for AR [1] rather closely.
Naturally, all the usual caveats about word meanings changing over time apply, but this one is a pet peeve of mine.
[1] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
[2] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
[3] https://trends.google.com/trends/explore?date=all&geo=US&q=H...
[4] https://trends.google.com/trends/explore?date=all&geo=US&q=%...
These early MR references use the term to mean a spectrum from "Augmented Reality" to "Augmented Virtuality". The AV term never really caught on, and the Wikipedia link for AV [1] actually just redirects to their page for MR [2]. I can't think of many notable examples of AV at the moment, though something like "Mario Kart Live: Home_Circuit" [3], as mentioned on the Wikipedia page for MR [2], might fit the bill, in the sense that reality is being used to augment the game. Uses of haptic feedback in VR might also count as AV.
This usage of MR has nothing to do with how Microsoft began using the term. It's almost as if they embraced an academic term for the sake of marketing, extended the meaning, and successfully extinguished the original academic definition.
[1] https://en.wikipedia.org/wiki/Augmented_virtuality
[2] https://en.wikipedia.org/wiki/Mixed_reality#Augmented_virtua...
[3] https://en.wikipedia.org/wiki/Mario_Kart_Live:_Home_Circuit
(Assuming by "mix depth to objects with the real world" you mean "having virtual objects alongside real objects with convincing depth alignment")
People are now using "MR" to mean something distinct from AR but GP and myself are arguing that the term AR was broad enough to encompass passthrough, overlay and any future developments.
If someone built a VR headset that used small projectors rather than flat panel displays, we wouldn't need to invent a new term for it. The AR <> VR spectrum was all we needed and MR is just muddle on top of that.
Alternatively, if you are going to just stick to one hyper general term then you can't really complain that product X is bad at Y because it doesn't do every single thing the general term Y encompasses. Language is fluid for a reason - it adapts to our needs to define and distinguish things. In this case, the Quest Pro does a subset of things quite well, while completely leaving other aspects of AR on the table. Whatever we refer to that subset as, it's not really valid to then say it's generically bad just because we don't have good words to describe what it does do.
I was a bit sloppy in that regards. I was more worried about what was going on. But when combined with the word "passthrough" to see the real world, I tend to drop back/slip to the term AR passthrough which is what this type of thing was called for years. MR, as I remember it, is a more recent term used to distinguish different types of AR. Then things flipped and XR (=AR/VR/MR) was used to mean was AR used to mean.
My guess is that late in the program, the importance of passthrough was elevated, perhaps in response to Apple rumors, but they were stuck with the hardware that was in the pipeline. The passthrough available is a big improvement over the Quest 2 for use in seeing the grouse-level surroundings, but not up to regular use in full Mixed Reality.
There are probably decades of "optimizations" left to do.
That's literally the same thing in terms of hardware. If you want to bring virtual objects in the real world, and not have weird z-sorting issue all over the place, you need to have an understanding of the real world, which in terms gives you the possibility to do augmentation.
Beside, MR is just a confusingly overloaded term these days. It can refer to:
* Microsoft's Mixed Reality brand of VR headsets (that completely lack any AR/MR features)
* filming people playing VR games in third person and mixing the game footage into it with LIV or similar software
* doing AR with VR headsets and pass-through cameras, instead of see-through optics
> But why only a single color pass through camera?
Pico4 makes the same mistake. I really don't get it. Lenovo Mirage Solo back in 2018 used two front facing cameras at a proper IPD, and while the image is still quite low resolution, having actual 3D with zero distortion made the thing feel like wearing actual glasses, not like looking at a weird camera feed. It's a ginormous quality improvement for very little extra effort. Using pass-through on that thing is still the only time I could literally forget I was wearing a headset.
I am fairly confused by the low res cameras used in the QPro. I have to assume Meta were just heavily constrained by the need for them to do double duty as tracking cameras and pass through cameras, so higher quality cameras just weren't available. But why only a single color pass through camera? This really doesn't make much sense other than if you allow that the processor on the Quest Pro is so limited that even if it had both feeds it probably couldn't process them.
I think it's actually great news that there is so much optimization left to do. Seems like they should be able to improve it a lot over time, and in the meantime it's pretty adequate to enable exploration of some unique experiences.
edit: VR => MR