I'm kind of impressed that a kernel panic doesn't just result in a black screen and that passthrough still works.

I’m not super informed here, but I believe the AR stuff, including eye tracking, takes place on a coprocessor. The OS on the M2 chip could die and you’d still have passthrough.

Sure it helps but does it _have_ to be on a separate chip? In ye olden days when everything else had crashed and was frozen, skype calls still kept working

Historically, it's generally been cheaper and easier to have two chips than to persuade a consumer or server OS to provide reliable time guarantees.

Hypervisors like that already exist and are needed by cloud providers.

Well, even in that context you have AWS for example doing control plane operations on a different SoC - https://docs.aws.amazon.com/whitepapers/latest/security-desi...

I think it's more the fact that the M2 chip is used across multiple product lines that don't need the unique capabilities of the R1 chip.

I think it makes sense. The parts of the system that need to respond within certain time windows because of physics run in the R1 (presumably, “realtime 1”). Everything else is just an iPad, which currently runs on the M2.

Valve Index is similiar. I've had my whole desktop freeze but the spatial orientation is still drawing within the VR as the headset is moving albiet in a frozen state.

The Valve Index does not do reprojection on the device. It does not even do tracking on the device.

It does it on the PC?

SteamVR’s tracking page[1] explains it a little. The HMD is really just displays, radios to talk the controllers (standalone trackers will just have a dongle, runs the same firmware iirc), and its own chip to talk to its own tracking sensors. All the tracking stuff amounts to timing information which is screamed over USB to the PC which does all the math to figure out where you are.

A bulk of this (really interesting!) information can be found on their HDK documentation, but that requires you to sign up with Steamworks.

[1]: https://partner.steamgames.com/vrlicensing

Yes, the runtime handles interpreting the sensor data collected by the device to compute the position and orientation they are in space. The compositor will reproject the next frame it sends to the headset if the application producing the frames does not submit frames before the deadline.

Yep, R1 chip.

It's a pretty great redundancy design though. Imagine someone driving or doing a life-critical task (augmented surgery?) on the Vision Pro and losing passthrough at a critical moment. And yes, there are already videos of people driving their car while wearing their headset (https://twitter.com/lentinidante/status/1753549370568368224_).

I like to imagine a possible future where Vision Pro can help you see your blind spots while driving a vehicle.

No shiny AR or "spatial computing" distractions. Just visibility.

That would be incredibly useful, and cool.

You people are fucking insane... You really want people wearing this crap driving cars and doing mission critical things??

Christ the people on the website are just another level.

I mean, it either works well enough for that to be a good idea, or it doesn’t. There are already technologies that are crucial for people to do mission critical things, and if they work well, then of course I want people using them, even if technology might feel really scary sometimes!

I'm assuming that any future where Apple actually supports driving with the headset on would have to be a future where all the ethical and safety concerns have been worked out.

The people driving today with the headset (even for the memes) are the insane ones. It does not have the FOV for you to be fully aware of the road (I've used one in-store). I've also heard that even 12ms is apparently still too much latency for a driver to react promptly to emergencies.

> any future where Apple actually supports driving with the headset on

As another user of the road do I get a say in this as well besides Apple and the moron with their headset on?

This is one of the dumbest things that you could possibly do, a heads up display I can get behind assuming it is very much limited in how much information can be displayed but a device that utterly obscures your vision, removes your peripheral vision and replaces it with a screen that isn't necessarily pointing in the direction that your head is pointing in whilst driving a vehicle sounds like an excellent way to have a serious accident. I do hope it is a one sided one, this is Darwin award level stuff.

By "safety concerns" I do mean the existing hardware issues with the headset as well, such as the 12ms latency, bad camera quality especially at night, extremely low FOV (should be 180+ degrees at least), instant shutdown when power is removed, screens that go dark when power is removed, etc.

Your glass windshield doesn't need a source of power to stay transparent, but Vision Pro does.

I absolutely do not believe that the current iteration of the product should ever be used while operating a moving vehicle. You are right to call it dumb.

Peripheral vision extends widely outside of the angle that a typical display affords and moving your head enough to see the mirrors in the car will bring a much larger area to the left of you in view. You can't simulate that with a display in front of your head to the degree necessary that it makes up for the losses. I think this idea is a complete non-starter. Note that 'heads up' displays improve safety because you no longer have to change your focus for nearby and far while driving (dashboard dials, navigation, vs what's happening in front of you). This does the exact opposite, it removes your ability to see what is going on and then replaces that with a bunch of bits. You're a software glitch away from an accident and with all respect to my fellow software developers I don't consider any of us competent enough to get that right to the point that you can potentially unleash that onto a billion drivers and not expect utter carnage.

> You're a software glitch away from an accident

Software or hardware glitch. Or power glitch.

R1 runs an RTOS, though, passthrough continues to run even if the main processor (the M2) completely crashes. So it already seems to try to fail safe. I can't imagine that's by accident. Maybe it's not that they're going to use the Vision Pro in these situations, but that maybe they might use the R1 or a future generation of it for their self-driving cars.

“want” doesn’t need to be involved.

running from the inevitable only makes you tired.

Chill dude we like the technology.

I think this is just a side effect of the fact the Darwin kernel isn't sufficiently low latency to ensure pass through feels 'real' enough.

Therefore the data path from camera to screen has to entirely happen without the kernels involvement.

I feel like the small passthrough delay was slightly noticeable upon first use, but with motion blur it was easy to get used to, and now I don't notice it whatsoever. If I had to guess, it feels like it's under 10ms.

interesting pattern arising where the technical theory is getting push-back from real world experience.

> If I had to guess, it feels like it’s under 10ms.

too many miss the value in such a statement.

“feel.”

subjective, but also sublime.

it doesn’t need to meet a metric, it needs to match a vibe. the metric is an approximation, as backwards as it might seem.

RealityDevice running on xrOS.

That only shows a kernel panic. An exploit would prevent the kernel from crashing.

How can malware prevent a panic? Its process is priority over pid1? I'm very ignorant to this stuff. Wouldn't the kernel know it's being manipulated, like a startup/JIT checksum or something?

The general idea is that an exploitable remote code execution bug is some specific circumstances where the processor starts to execute whatever is in some wrong address which can be manipulated by an attacker.

If the bug is successfully exploited, the attacker arranges so that everything is just so that device runs "as if normal" but executes whatever code attacker decided to be executed; but if the bug is triggered accidentally or without a working exploit, then it attempts to execute something random and that results in a kernel panic.

So the way how "malware prevents a kernel panic" is that it's a successful exploit if and only if it manages to ensure that valid (but attacker-controlled) code runs and actually works, and if it instead results in a kernel panic, well, that's not a working malware, unless all the attacker wanted was a denial of service exploit.

A kernel panic is caused by some serious issue in either the kernel or the hardware. In itself, it's relatively "harmless": it just means the OS stops doing anything and you need to hard reboot the machine (such as by cutting power).

However, if the kernel panic is caused by a software bug (rather than something like a broken RAM stick), then the bug which causes the kernel panic might also be exploitable by malicious code to do things like arbitrary kernel level code execution. For example, if the problem is that the kernel somehow allows the user to change a piece of memory which the kernel interprets as a function pointer, then the result of carelessly changing that piece of memory will cause the kernel to jump into some part of memory that's not valid code, causing a kernel panic; however, a very clever attacker may pick a jump address which causes the kernel to keep executing without crashing but doing something which the developer of the kernel didn't intend. Maybe the attacker can cause the kernel to jump to a location in memory which the attacker has control over, which would mean that the attacker can run arbitrary code in kernel mode.

This stuff is all very complicated and very interesting, I recommend reading up on it if it interests you.

And no, the kernel can't really check if it's being manipulated. A checksum at startup wouldn't do it since these sorts of exploits don't tend to involve changing the kernel bytes on disk, but rather tricking the kernel into doing something it's not supposed to. There are various mechanisms to try to make bugs unexploitable (such as address space layout randomisation), but fundamentally, you can't perfectly protect against potential bugs.

Thanks! Yeah I really need to work on my low level security knowledge this stuff is fascinating when it isn't presented in the form of a crypto algorithim that is heirogylphics to me.