News for hackers. Stuff that matters.
Re:Meta Quest Pro – Bad AR Passthrough by Zee2 on Tuesday January 03, 2023 @09:20PM (#34241128)
As someone close to this in the industry, I definitely agree.

I also really don't know why he decided to deemphasize the perspective and depth correctness so much. He mentions it here:

> In this case, they were willing to sacrifice image quality to try to make the position of things in the real world agree with where virtual objects appear. To some degree, they have accomplished this goal. But the image quality and level of distortion, particularly of “close things,” which includes the user’s hands, is so bad that it seems like a pyrrhic victory.

I don't think this is even close to capturing how important depth and perspective correct passthrough is.

Reprojecting the passthrough image onto a 3D representation of the world mesh to reconstruct a perspective-correct view is the difference between a novelty that quickly gives people headaches and something that people can actually wear and look through for an extended period of time.

Varjo, as a counterexample, uses incredibly high-resolution cameras for their passthrough. The image quality is excellent, text is readable, contrast is good, etc. However, they make no effort to reproject their passthrough in terms of depth reconstruction. The result is a passthrough image that is very sharp, but is instantly, painfully, nauseatingly uncomfortable when walking around or looking at closeup objects alongside a distant background.

The importance of depth-correct passthrough reprojection (essentially, spacewarp using the depth info of the scene reconstruction mesh) absolutely cannot be understated and is a make or break for general adoption of any MR device. Karl is doing the industry a disservice with this article.

Re:Meta Quest Pro – Bad AR Passthrough by zmmmmm on Tuesday January 03, 2023 @09:30PM (#34241216)
yes I think the success of the reprojection is a much under appreciated achievement. The author repeatedly questions their claim of "seamless" but its one of the most stunning aspects that you can, for example, see your real arm outside the headset exactly map onto your virtual arm and hand inside it, as do other objects that bridge from external reality to the virtual reality view. This "seamless" nature of it is really impressive when you consider that the whole view is actually being completely reprojected and not just displayed natively from the camera feeds.
Re:Meta Quest Pro – Bad AR Passthrough by Zee2 on Tuesday January 03, 2023 @09:36PM (#34241267)
Yeah, I really don't think people understand how critical the reprojection is. A lot of folks, apparently Karl Guttag included, think that the resolution of the passthrough is the most important factor for usefulness. And/or that the resolution of the cameras is even directly related to the quality of the passthrough to begin with! The pixel count is just one of the dozens of quality metrics, and is arguably actually one of the less important ones.

I also can't tell if KGuttag believes that you can simply pass the video feed through to the user and not have an absolute usability disaster. His paragraph about the "pyrrhic victory" sort of implies that not preferring depth-correct scene reprojection was an option. In my experience, that is absolutely non-negotiable in terms of shipping a device that's usable to the majority of consumers.

Another reason why folks also don't understand why passthrough is so energy intensive. You're also running an entire additional pass of steresocopic spacewarp on what is essentially four different camera feeds, on an incredibly high resolution scene reconstruction/SLAM mesh.

Re:Meta Quest Pro – Bad AR Passthrough by ncr100 on Wednesday January 04, 2023 @12:28AM (#34242355)
Post to his blog if you want an answer - he responds
Re:Meta Quest Pro – Bad AR Passthrough by wvenable on Tuesday January 03, 2023 @11:50PM (#34242148)
In a regular Quest 2 (not the pro as discussed here) you can use pass-through and actually reach out and grab a glass of water and drink it. It's super impressive and actually makes the pass-through useful for the things that you use pass-through for.

Would it be nice to be able to actually read text and not have distortion when objects get too close? Yes, of course. But it's not necessary.

Re:Meta Quest Pro – Bad AR Passthrough by Zee2 on Wednesday January 04, 2023 @01:10AM (#34242585)
Exactly. Reading text or having 20/20 vision is so, so, so not the most important part of passthrough! Comfort, accuracy when reaching out and grabbing things, walking/running, avoiding obstacles on the ground, pets/humans, etc. Non-depth-correct passthrough risks all of these things.
Re:Meta Quest Pro – Bad AR Passthrough by blensor on Wednesday January 04, 2023 @02:06AM (#34242902)
We are actively using passthrough in VRWorkout (to be aware of your surroundings while jumping and knee high running) and I have to say that even the low quality Quest 2 passthrough feels magical for that [1]. If you do look at 0:20 you can see that it really shines when the virtual hands interact with real objects

[1] https://youtu.be/DDwmXsvD-NI

Re:Meta Quest Pro – Bad AR Passthrough by cokeandpepsi on Wednesday January 04, 2023 @03:01AM (#34243230)
Yeah I've been really enjoying the passthough as the homescreen background feature they added, it also works on WebXR pages too now which has been fun for the small experiments/toys I like to make on my downtime
Re:Meta Quest Pro – Bad AR Passthrough by nomel on Wednesday January 04, 2023 @01:35AM (#34242749)
I’ll admit, I have completely cooked breakfast (eggs and toast, rummaging, flipping and all) with the Quest Pro passthrough, while watching a video on a giant virtual screen floating in my kitchen. The depth accuracy is very impressive.
Re:Meta Quest Pro – Bad AR Passthrough by geokon on Wednesday January 04, 2023 @02:18AM (#34242970)
Just so I understand correctly - the fact that the cameras are places a couple of inches in front of where you're actual eyeballs are causes a lot of discomfort? And so you need to warp the camera feed to look like it should from your eyeball's perspective?
Re:Meta Quest Pro – Bad AR Passthrough by andybak on Wednesday January 04, 2023 @03:55AM (#34243486)
I'm not sure if it's entirely due to the camera position or if there are other factors (e.g. differences between the camera optics and your eye's) but I can confirm that using a device without reprojection results in quite extreme distortion for objects closer than an arm's length.

Which unfortunately includes your hands.

Re:Meta Quest Pro – Bad AR Passthrough by ghusbands on Wednesday January 04, 2023 @09:22AM (#34245765)
There's also the distance between the eyes not matching that of the cameras. And the fact that the 3d scene that isn't from the cameras does not have your eyes in the same position.