It seems obvious to me that there's a progression. Back in Apple II, C64, DOS days computers generally ran a single app at a time.

Windowing systems were invented allowing multiple apps to share the same screen. This is what we've had for 30-40 years.

For AR though we need multiple apps to share the same 3D space all running at the same time just like I have 14 apps running on my laptop right now.

I need my mapping app drawing the path to my destination. I need my transportation app showing me the next bus/train coming. I need my virtual pet app adding my virtual pet around my feet. I need my social networking app to show me my friends info next to them in me view. I need my dating app highlighting people that are potential matches.

All of this has to happen simultaneously and at 90fps. That will require an entirely new OS and possibly new hardware (dual GPUs, ... or pre-emptable GPUs) one for compositing the 3D world at 90fps and one for apps so if app takes too long to generate it's portion of the scene the entire OS doesn't start running low. An old OS that only runs a single 3D app at a time and who's apps don't share the same 3D space will be a huge hindrance to AR being truly revolutionary.

I'm not saying Magic Leap's OS is that OS but that OS is not MacOS/iOS/Windows/Linux/Android

One of those is not like the others: “Linux” is not a graphical environment. If you had said “X11” or “Wayland” (or even “Gnome” or “KDE”, etc.), I would have agreed with you, but “Linux” could perfectly well be the base of an entirely new 3D-based application environment.

This is something that I've given some practical thought to. The approach that I considered was to treat each application process as a client to a '3D space' service through which it could add and modify defined geometry.

Failure/slowness of any given application would leave the existing application geometry in-situ within the 3D environment and avoid the most jarring extremes of user experience. I imagined that the service interface itself would mandate the availability of simplified geometries together with meta information. This would allow the service to appropriately degrade the 3D rendered environment to maintain high framerates.

I think this approach could be effective and would not require a new OS.

Examples:

In my state we have a govt funded web/native app that provides the fuel prices at all the service stations around you. If there was a mapping extension you could turn on, it would show this directly within the mapping app, and also an indicator "cheapest fuel within X km is at Y location" when you're in navigation mode.

In camera view with an extension turned on, provide object detection similar to how QR codes are recognised.

Apps which crash or otherwise stop rendering trigger a fade out to the white-room VR space, whereas frame-skips are handled by hard drops (so the effect is like a teleport to avoid motion sickness).

Like, Valve have really done an incredible job on the user application interface there.

This is already true with even the oldest and most rudimentary AR/VR systems, eg, Gear VR. The interface and head tracking is 90fps all the time. If the app is slow, the viewport will not freeze. Just it will be stuck on one frame.

Sure, it might require special hardware and software (even your screen has it's own software and processing that it is doing on the signals coming from your desktop), and at the end of the day the OS would need to change and evolve to handle this new paradigm, but it's not impossible to imagine a system where the "PC" does what it does now, and just tells the "VR" sub-processor/sub-machine where to put the rendered "window".

Already there are some VR apps that basically wrap your normal "windows" desktop in almost a cylinder around you and you can move screens around on it as much as you want with current VR headsets.

MagicLeap is "An OS for VR/AR stuff that you make apps for", and that's not what I'm personally looking for regardless of how many other OSs it will integrate with.

My "perfect" AR system is one where I can run Slack, Chrome, my code editor, a few terminal windows, and more, all unchanged. And MagicLeap looks like it will never be that. HoloLens will, even if only on windows.

Now I might be wrong, and MagicLeap might knock it out of the park and become a new "category" of device like phone apps did, but I still can't use a phone OS to do real work outside of a few choice situations, and I REALLY want a VR/AR device that lets me do real work.

I wish them the best, and while I get annoyed at every article that talks about it handwaving away the tech and calling it "magic", I hope they do well, because more competition is always a good thing, but from what I've seen they aren't making a device for me.

It may come from ML 4 generations and a decade from now, it could come from google, apple, microsoft, or some as-yet unknown company... but it will come and it'll change everything.

If by computing you mean video/game consumption then yeah sure, but for pretty much every other application (programming, editing, reading, etc) this is a poor medium.

Games and stuff are nice, but everyone I've known that uses a vive or oculus gets tired of needing to move around in VR pretty fast.

But being able to get rid of the 3-4 screens on my desk, and replace them with a headset that I wear? That I can put windows everywhere? Behind me, far left, even throw one up above me to glance at? That's HUGE imo. And the cost can quickly make sense.

I'm in the market for a new set of screens for my machine. I want to go to 4k. I'm looking at realistically $1500 for new screens. If something like Magic Leap or hololens was only 2x more than that and could get even "1080p equivalent" resolution compared to my screens, i'd happily drop the extra cash right now.

Yes, it's still not at that point as far as I can tell, but I don't think needing to increase resolution a bit is going to be the thing that ends up making VR/AR "impossible".

Are you sure about this?

Because that's the idea i'm envisioning. Being able to put papers or applications on a virtual whiteboard that completely encircles you. Move them to where you want, make them "fullscreen" (full-view?), maybe even be able to move them forward and backwards in focus and "position". Hell, with some extra work, we could even put them on surfaces around the room. Throw a monitoring program on the wall next to my window, "project" keybindings on top of my keyboard's keys when i'm pressing parts of a combo, etc...

Hell, even if you hate the idea of an "augmented reality desktop", you could just project "fake" screens and use them the same as you normally do.

I mean it's not like our current tools just go away and I'd start craning my neck to look up and to the left as far as possible for 9 hours a day... You could still have tools like "multiple desktops", and change/move them. But now I have the option of putting my task manager above me at like a 60 degree angle straight ahead and can glance up at it when needed.

What in the world do you put on those other seven screens?

I can't think of a situation where I'd need to focus on all the things you list simultaneously – even just having e-mail on the screen while I'm trying to code is distracting – so I'd rather have them hidden. Sure, I'll have a couple code windows and documentation or logs open, but that fits with room to spare on a single monitor.

> offline documentation (i.e., not in a web browser)

This deeply confuses me. Why does it matter if the documentation is in a browser or not?

man and info come distraction-free out of the box.

(And before you ask: Yes, I use one laptop screen for all my work.)

The only people I know using 4+ screens are monitoring Satellites or industrial processing equipment.

With this, you could just float individual windows around you. Quick splitting and navigating between splits in Vim is already one of the features of this editor I use the most - and if I could do that in 3d, leaving small splits all around me and moving them around, it would be an immense boost in productivity.

Of course you would, that's like saying "if something like a Tesla could operate completely autonomously in all weather and road conditions I'd happily drop the extra cash right now!". Unfortunately, this is a "hard" problem and while I'm not saying it won't happen, we are nowhere near anything that can render 1080p VR monitors... 4k VR monitors are unlikely to be possible (in a consumer product) within the next 20 years, and you can definitely be sure that anything like that will not be wireless.

Think about where this will be in 10-20 years, not where it is now. It will be like putting a pair of glasses on, not a huge headset. Once your workspace can exist in 3 dimensions, new paradigms will emerge for arranging a "desktop" that we can't even envisage right now.

Huh? I'd love to get rid of the chunky, heavy, power-hungry screen. Read lying in bed? Check! Program on an escalator? Check! Watch a big-screen movie while eating in McDonalds? Check!

Anyone who's tried VR knows what that means - 1920x1080 is not enough useful lines for serious work with text, it's just barely enough for gaming.

I think you are right about higher resolution being needed for real "work", but since this doesn't run a "real" OS either, i'm not sure how much work you could get done on it anyway.

I don't think wearing headsets is a practical way to do everyday computing and it can even be bad for your eyes.

With a monitor you can look away, look out the window, etc. but with a headset the screen is always in front of your eyes, so you have to take the whole thing off if you want to give your eyes some rest from the screen.

It may be good for shorter special tasks like gaming, vr assisted surgery and such, but it doesn't seem a good match for programming, for example, which you do all day.

Check out a Hololens if you ever get a chance. You can choose if the object you create is always in front of you or 'stuck' in a fixed point in (virtual) space. If I place a 'screen' on a wall it will stay exactly where I put it no matter how I move around.

With a good AR headset, that focal point can be adjusted and moved around (possibly even in realtime if you want, move a screen to be "further away from you"). Not to mention that when you aren't looking at something that the headset is "generating", it's just basically a clear piece of plastic that you can see through. They are basically bad sunglasses.

You can look away from a "projected" screen just as easily as you can look away from a physical screen.

Magic Leap has been under a lot of pressure to prove that they are not vapor ware. Which, is why I assume that they chose to launch this now. Unfortunately, their vision and their product are obviously still miles apart and the content is a little underwhelming right now.

Probably the best way to look at this would be as SDK for early adopters. The first version of the product that doesn't require a ton of NDAs to be signed to even be allowed in the same room with it. However, the price point is such that this is clearly not intended to be a mass market product.

For a mass market product the price could actually be fine but then the experience needs to match the price. People spend way more on fancy TVs, high end laptops, gaming hardware, etc. For that to happen there needs to be a supply of compelling content.

The UX for AR is going to have to be substantially different from whatever your current OS does. Overlaying a lot of 2D windows in your peripheral vision doesn't really make a lot of sense. Sure, it's a neat trick but probably not that practical. Similarly a lot of mobile UX is inherently 2D currently and actually optimized for tiny screens manipulated by clumsy fingers. Slapping some AR on that is not going to be very compelling.

With VR this was more straightforward since there were a lot of 3D games that with a little work can be awesome in VR. So far gaming is the killer use case for VR. People spend money on the latest VR kit so they can play all the awesome games that are out there and immerse themselves.

On the AR front the most popular thing so far has been pokemon go and a range of me too style clones of that. That was only possible because of a critical mass of cheap, camera equipped mobile phones. With AR we need new content specifically made for AR. Most VR content does not port over well to AR since the whole point of that content is full immersion and the whole point of AR is not replacing reality but enhancing it.

Clearly magic leaps strategy for this is to have their own OS, app store, devices, and platform. This sounds like a risky strategy since clearly all their marketing is about the qualities of the device and how cool the technology powering that is. It's significant how they are not talking about the rest of the ecosystem.

From what I've heard via a friend of a friend on the hololens team, MS is putting a lot of resources into building out that with a lot of cool R&D in the pipeline. MS of course has a long history of cool R&D projects that ultimately never turned into products but this looks like they are pretty serious about it.

Unfortunately it doesn’t matter what you (the consumer) wants. Do you want a single search engine that dominates 90+% of the market and serves shitty ads to you all over the internet? Do you want a single walled garden for your social presence online that captures/sells your data and serves you shitty ads? Do you want 2 main mobile OS’s that tightly control their platforms and the apps available to you?

I will say this though...people saying they want a screen strapped to their face, as you did, is an advertisers wet dream and these tech companies can’t wait to sell you that screen and serve you those ads.

I don't care if i'm tethered to a $3000 desktop PC, because I have an office anyway and it's where i'm going to work.

Presumably you're still going to need a keyboard.