I'm quite certain they're still trying to solve the hard see-through AR problems, and are hoping to release a future Vision headset with a true see-through display.

But otherwise I agree, it makes sense for them to focus on what they can do best with current tech as a stopgap. With current see-through HMD tech, AR ends up incredibly disappointing. (See also: Hololens & Magic Leap's limited FOV)

My understanding is that occlusion in "see through" AR is an unsolved problem, everything looks ghostly and somewhat semitransparent. Until someone solves that I suspect the re-projection method is the only viable option.

Yes "projecting black" is effectively impossible - every once in a while someone from Magic Leap would claim they had solved it but everyone knew it was bullshit

I'm sorta surprised by this. Don't liquid crystal displays in laptops do exactly that?: LCDs are a controllably transparent array that is overlaid on a uniform backlight. So why can't one create a set of glasses where the lens are an LCD array without the backlight?

An LCD simply turns "off" to display "black" - so the emissions stop (relative to surrounding pixel emissions)

With a projector, you can't "throw" nothing (aka black). As a result "projected black" is simply lack of projection.

In the case of a translucent or transparent reflection or waveguide surface - which is what the projection reflects off of - "black" is whatever the darkest part of the surface is. In effect whatever else is emitting from the surface that you're looking at will change the depth of "black" you get.

This is why the Hololens and other see through AR devices are always tinted, to set a higher threshold for "black" than the surrounding unaided view.

It's more accurate to say than an LCD allows light to pass through a 100% polarizing filter. The are three layers of polarizing material. The two outer layers are at right-angle polarizations to each other and normally would be completely opaque on their own. When power is applied to the liquid crystal, it twists the crystal's polarization to be at a 45° angle to the other two layers, which then permits about 25% of any incident light to pass through.

An optically transparent waveguide display can use an LCD layer to block light coming through the front and then not render graphics on that area of the display. It will be opaque black at that point (though rather fuzzy around the edges, as the LCD won't be in focus).

Magic Leap 2 actually employs this technique. It's... a lot like the rest of the device: a good idea on paper.

That’s a fair response, though I hope you’d agree that in the context of discussing pass through Vs see through “black” the majority of use cases are indeed fully occluding/lit LCDs near eye and not ML style lenses.

But you were talking about the Magic Leap, replying to someone talking about waveguides...

Magic Leap.. How are they still around? And have a valuation in the billions?

I don't know. I mean, their newest device is better than the HoloLens 2. But like, that's just a relative statement. Waveguide displays are still objectively dogshit.

HoloLens stopped being a focus of Microsoft and it shows in the product

https://tecl.ink/2023/06/apples-vision-pro-wearable-headset-...

You have to be looking through an LCD for that to work, but all existing AR HMDs are head mounted projectors with transparent mirrors, with no means to control, say, pixel level local reflectivity of the mirror. The mirror is just a flat mirror.

If you forgo the projector part and replace the mirror with a transparent LCD, it’s just too close to your eyes and you can’t see anything. If you add a microlens array to the LCD, now the LCD might come to focus but background becomes way too far-focused, and you can’t see anything either.

If we were on an Enterprise-D, I guess I could just ask replicator for a passive illuminated metamaterial light field image combiner with integrated processing than runs on bus power from DisplayPort input, but we are not there yet.

So, for now, our AR HMDs can only brighten pixels against backgrounds.

I think I understand. The intended idea was, e.g, a pair of "glasses" where in front of each eye are two layers, (1) a partial transparent mirror and (2) an LCD array, so that you could darken a pixel to block light coming through the mirror, and you would ideally only see light from the projector that was being bounced off the mirror and entering your eye. But you're pointing out that those LCD pixels won't be at the same focal depth as the objects out in the world you're looking at. Thank you.

This is a concise way of describing the problem I think

I work in the space. Occlusion only matters if it matters. "Ghostly" information display is suitable for a shockingly wide range of tasks.

While that's true, I believe these (mass market) devices need to be general purpose, and so need to cover the use cases where occlusion is necessary.

This is also what I intuited. Can you give some examples of things that require “projected black” , especially those that couldn’t be solved by using a darkened room?

Right, I think it's going to come eventually but it's really really really hard

They are still asking suppliers to solve the technology, yes.

Hopefully we are not forgetting about Bosch's Retinal Projection which solves many problems with lensing HMD AR. If I was a betting man I would say Bosch came up with this tech almost specifically for Apple to integrate it into interactive glasses.

Yes...

https://tecl.ink/2023/06/apples-vision-pro-wearable-headset-...

Just curious - why is it so hard?

Off the top of my head:

1. Field of view is limited to existing optics miniaturization

2. Subtractive shading (rendering black) might not be solvable

3. Variable focus objects in the same scene requires projecting n>2 significantly different wavefronts - not solved how to do this with a single vibrating element

Do you know why an LCD panel can't be used to block light?

Which pixels need to be darkened to shade off an object depends on the distance to that object, and will also block the light coming from other objects at other distances. It's very inconvenient.

The LCD panel is too close and out of focus.

For example, close one eye and hold the tip of a pen about an inch in front the other and you'll see that it doesn't actually block any of the world.

The Magic Leap 2 does. It works ok. It's not as completely useless as armchair quarterbacks on line would have you believe. But it's also not the greatest thing. Objects are a little fuzzy around the edges and friend on the accuracy of the object surface detection. So moving objects can lag.

I'm not aware of all the details around the technical complications, but from a physics perspective, you can't make something darker and more opaque by adding more light. Therefore, an AR headset needs to be able to at least partially block light in order to make convincing images. It seems a lot easier just to go with Apple's approach of blocking all light rather than try to develop tech that will only selectively block the light behind the AR objects while allowing other light through.

The blocking all light approach also allows you to hide other potential weaknesses of a device. For example, a lower field of view is much more distracting in a pass-through AR device as you still have your full peripheral vision. VR devices will generally black out the light outside of the FOV making it easier to ignore.

> you can't make something darker and more opaque by adding more light

I mean... theoretically you can if you can match photon for photon with the inverse phase. Difficult, I know :)

It's one of those "small, low energy, bright, pick 1-2, definitely not 3" type situations.

Moore's law works great for semiconductors, but Maxwell doesn't negotiate ;)

Nobody has a good way to draw dark while keeping it in focus.

another follow-up, slightly tangent question: how does F-35's helmet do that without blocking all the lights?

(Caveat: I don't know a lot about pass-through HMD AR, and I assume you're incredibly smart and that the patent is innovative.)

> …it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology…

I understand the framing as "compensatory technology", but is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. But a 100% digital vision pipeline seems like it could unlock interesting capabilities like "night vision", new ways of highlighting interesting objects, etc.

In theory the wavefront emulator is the simplest display - all of the "pixels" are rendered in your brain so you don't have to build an actual "display"

HOWEVER if someone can get the input -> photon production pipeline to be less than ~10ms, then that does solve a lot of the rendering issues, however it doesn't solve all of the other long term problems that come with that amount of hardware - including weight and complexity.

That said, there's a lot of known-unknowns that need to be solved, for example I don't have a solution for micropiezo resonance issues that I'm sure will crop up.

I think the most prominent attempt at see-through AR was the Microsoft HoloLens. But if you've actually tried the HoloLens, the Field of View is atrociously, tragically small. The first HoloLens had a field of view of 30°*17.5°. The second HoloLens improved to 43°*29°, but it's still best described as "cramped." Couple that with almost all of the compute budget for the device going into vision processing and having very little compute left for actually running apps (the first HoloLens having a 1Ghz Intel Atom from 2015, the second a superior... Snapdragon 850).

The other problem, of course, is that nothing can be truly solidly-colored. Everything has some opacity - which, combined with the FOV issue, is why HoloLens was never marketed as having anything to do with VR.

The vision processing was done with a separate SoC. Your applications were not contending for compute time with the vision processing. That's just how anemic the CPU was.

Apple’s Vision Pro launch shows what Microsoft could have truly achieved with HoloLens

https://tecl.ink/2023/06/apples-vision-pro-wearable-headset-...

Agreed. In fact, this approach is also likely better for military applications which I hope they explore.

Currently, NVGs that are fielded by soldiers are already displaying images in a way that's not pass-through (using classic image intensification tubes). Something like the Apple vision headset in a lighter and more durable form factor would allow for eg; fusion imagery (fusing visible, thermal, and night vision).

It's not in Apple's interests to get into military contracting because of the potential to damage their brand, as well as avoiding the geopolitical tensions that would invite.

Apple won't. But as soon as these headsets hit the market manufacturers will be tearing them apart and reverse engineering them.

$4k/unit is nothing when it comes to military hardware. There's also no need for the outward facing display, etc.

If anything, I wouldn't be surprised if Hololens goes in this direction. And the DoD already has contracts around the hololens.

I’ve always wanted an NVG to play with, and from watching firearms nerd rants I’ve learned:

- real NVG costs way above my budget, and all comes with defects. Realistically the tubes in civilian markets are all defective returns.

- even real NVGs can’t work in complete darkness, such as in a closet; moonlight or starlight is needed.

- real NVGs on firearms are often used in conjunction with an IR laser and a flashlight for aiming; advantage is solely that you have to have a goggle to see it, not that you can operate without making any emissions.

- partly due to above points, plain old CMOS cameras with modified 1” Sony sensors for better low light performance is starting to match practical usefulness of a real NVG(it’s Cold War tech so no wonder here). Latency is an issue, but more of “is an issue” and no longer a blocker.

… Vision Pro type of devices replacing NVGs indeed seem like a near future possibility.

I have a few NVGs. They're definitely expensive (a good setup is $10k+) but it's definitely not true that they're all defect returns. L3 image intensifier tubes that enter the civilian market are just as good as the military gets for the same product. For cheap (<$10k) NVGs that's a totally different story though.

They do work in complete darkness! You just need an IR illumination source. If not using a weapon mounted illuminator you can just use an IR flashlight. Thermal doesn't require external illumination but isn't anywhere near as useful for navigation. That said, good IR laser + illuminator weapon mounted systems are a few thousand bucks too.

My point about vision pro type devices is they can take a feed from real NVG tubes and potentially regular cameras and thermal and combine them. It'll be game changing when the tech is ready. It's not a matter of if but when.

> is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects.

I think they're different and not one better than the other. If I wanted to drive with an HMD on, or otherwise be in a situation where it could be deadly to have my sight turned off for even a second or to even have some lag or stutter in my eyesight, I'd much rather have a pass-through AR HMD. One's sense of sight seems much more reliable with it by its very nature. Such modes of failure are simply impossible with transparent plastic.

Your steering wheel and pedals are digital too. Airliner controls have been digital for decades. Factories have life or death safety systems that count on silicon and a software stack to respond quickly and without error.

Apple can make it possible if they wanted too.

These are all very different from the “HUD goes dark and my meat brain panics, causing me to get into an accident” scenario that GP is talking about.

There are multiple system running on silicon that have been made to not go dark in normal mode of operation. Some examples could be found at https://en.wikipedia.org/wiki/Safety-critical_system

I believe cars' steering are analog because it would too expensive to achieve the required level of reliability.

Planes are crazy expensive anyway,so fly-by-wire can be afforded.

Steering is analog. One of reasons why by-wire steering control did not succeed is user feedback, it was tried and failed. Brake pedal is analog too, just assisted and override-able. Gas pedal had been all digital for some time.

Also, it’s true that Airbus sidestick is fully electronic, but there’s a nuance in engine control being in the center on airplanes. Until automated, engine control on airliners was a task of a flight engineer, like it still is on ocean going ships. So fully digitalized engine control is a replacement to the FE, not necessarily an automation of what originally was a pilot’s task. Which means, I think, pushing a vehicle forward was never necessarily the responsibility of a pilot or a driver, though controlling where not to go is.

>Steering is analog.

This is changing rapidly. eg:

https://www.infinitiusa.com/infiniti-news/technology/direct-...

That was the failed attempt I was referring to, literally a decade ago. Toyota is doing a retry with bZ4X/Lexus RZ, and those three are the only known cars to have passed prototype stages with (front wheel)Steer-By-Wire.

Which is probably the point of the R1 and the real-time subsystem.

Yeah this is the most interesting bit in a way.

Wearing the quest in pass though mode is kinda cool for a bit, but the lag makes it horrible to use for any more than checking where a wall is.

If they really have made this feel instant that’s a massive deal on its own.

If 1T Apple are such magic tech wizards then why couldn't they make AirPower work?