The thing is, just because it's theoretically possible to interface "naturally", that doesn't mean it will feel natural. Much like interacting with voice-based assistants is stilted, constrained, and often awkward with misunderstandings and false starts.

Interactive VR will be no different. Either it will be very simple but entirely artificial, or it will be complex and "natural" but ultimately unsatisfying. We are nowhere near being able to implement anything approaching "reality".

But even if we were, I think you overstate the promise. "Having to learn a new UI" is exactly what you will always have to do to learn to do anything new, whether it's in a mobile app, on a computer, in VR, or in the real world with physical objects. Guitars, power tools, ping pong tables, pressure cookers... they all have new and different user interfaces.

Really?

Are you referring to existing technology, or some kind of Sci fi holodeck thing?

I can think of 2 broad categories of potential computer program. The first, old world computer programs, and I can't see flicking through virtual files in a virtual filling cabinet being an improvement over what we have now. And the 2nd category is 'real world' computer programs. Basically simulating real life. So ranging from VR watching the tv (what's the point) to white water rafting. Which I can see, but that's basically back to computer games. Which is the thing the VR always ends up being used for.

Yup. I can't see most software more usable with VR either. At some input fidelity, people would just recreate virtual keyboards.

It really does seem that a keyboard is a quite deep local maximum of human-computer interaction. Not a single other mode of HCI invented afterwards can match the combination of high versatility and high efficiency.

But virtual worlds, and the vision-oriented tasks (like e.g. 3D modelling, be it artistic, scientific, or engineering) - there I can see VR working much better than what we currently have.

At the same time, we are as close as we've ever been to human (or supra-human) spoken language parsing. Even if we have a ways to go on machine understanding / interpretation.

I can see (3D manual manipulation + verbal commands + weak-AI inference of intent) being a pretty compelling interface package.

For our primary coding purposes, I doubt we'll ever surpass keyboard.

But the counterargument would be that coding is a fairly esoteric set of criteria that we've specifically built around the keyboard (i.e. every character matters, no synonyms, etc). Were there an alternative... say, boilerplate code intelligently glued together by verbalized architecture directions? I wouldn't want to put a bet down. Rails convinced me that sometimes being smooth is more important than being best.

I'm not saying it's totally impossible to get rid of keyboards, but I feel this would require first significantly thinning down the amount of text that needs to be written, and the amount of abstract interactions with program that need to be done. The keyboard is very efficient in several criteria at the same time. Versus a 3D manipulation + verbal commands + weak AI, it's much more energy-efficient/ergonomic. Instead of waving your arms around in complex motions and talking a lot, you just have to move your fingers and wrists a little bit here and there. I know I can easily keep pounding at the keyboard for 8+ hours without much slowdown; I imagine that an interface of voice and direct manipulation would get one tired much quicker, if not mediated by another input device, or if the workload isn't reduced.

Now I was going to say that the trick is workload reduction, and that 'AI-glued boilerplate directed by voice' could just work by doing most of the task automatically, but then I realized: workload reduction doesn't work. As proved time and again in the last few centuries, as soon as you find yourself with extra free time due to some efficiency improvement, more work will be assigned to you to fill that time. You may not want it, your employer may not even want it, but someone over at a competing company thinks they can get ahead this way, and now everyone has to follow suit.

I think blade runner style emanators or Iron Man Jarvis is what you are looking at. Sure they won't help you do work, but they will do other things for you. You can call people, order things, watch movies, do other myriad other social networking stuff(Watch videos, see pictures).

For programmers of course its hard to work with those interfaces. But imagine a carpenter being able to order coffee, without having to deal with a screen.

That's the kind of utility we are talking about.

Yes for any sort of complicated input i.e. computer code, real world experience can not be used. On the other hand natural language processing almost good enough today for inputs with medium complexity like grocery list.

Real world interaction is obvious for certain fields where user does something that's comparable to a real world task. i.e. Loading ammunition, Surgery, Plumbing, Repair of various instruments.

For fields that don't have comparable real world representation I would imagine that we're again at the moment in computer science history where we have an opportunity to create a new desktop environment but in 3D space with certain well known natural gestures. For example transferring a file from my laptop to a smartphone doesn't have to be done with context menu, we can have a VR smartphone object and just drag file with our hands in that object lie we would do in real world if were to move a file from one place to another.

Instead of having to switch between different overlapping windows, we can have all windows laid out all around us in a comfortable zone. To go to particular window in focus we use certain gestures like pushing of pulling with hands.

Even if some gestures have to be created artificially, user would probably remember that better next time because of the muscle memory.

There's a really good discussion regarding this in Unity Unite 2016 (https://www.youtube.com/watch?v=HzPDTrWcjxc). In fact 2016 conference had a lot of good VR related discussions.

>I think that appeal of VR is to be able to use real world interaction techniques learned since birth to interact with computer programs without having to learn a new UI with another new app.

Anything complicated enough -- like most software uses are -- will remain so.

Some niche fields aside (e.g. games, robotic surgery, immersive education, exercise, and obviously porn, the niche field to end all niche fields) VR only makes using the computer more tedious (as you have to move your body). We have seen mostly the trend of disassociating with our bodies through computers...

Clip unavailable in my country, but upvoted for Summer Wars.

That said, I'm not sure what you mean by "a new Internet". I have the internet on my desktop, my laptop, my tablet, my phone, my TV, my game consoles, and even my lightbulbs. Do you just mean a new interaction paradigm? A new style of presentation? A new social network?

You're right, I mean the new style of presentation but in terms of everything presentable being connected in this system. (WWW)

Not sure about the blocking. I watched the movie a long time ago, in movie it's in the first 5 minutes introduction of OZ platform.