BTW, GPU is a problem, but we're expecting Frame Rate Amplification Technologies to solve the problem. Basically improved versions of Oculus Spacewarp that can do large framerate multiplication factors with zero parallax artifacts (unlike today).
I covered this topic near the bottom of a different article about the journey to 1000 Hz displays at https://www.blurbusters.com/1000hz-journey
The gist is that within five to ten years, we'll have many tricks to increase framerates with the same number of transistors, without needing to reduce detail levels or make textures/edges blurry, without input lag, and without interpolation artifacts.
When you're dealing with a head moving, and very brief slices of time, inertia plays a large role and allows for fairly accurate prediction. After rendering the frame they check head position again, update their prediction for head position at time of display, and move/warp the frame slightly to match. This does require rendering a slightly larger view.
I remember when Oculus cracked the 20 ms mark and got down into imperceptible lag, it was very exciting. They bragged at the time that their predictive models would let them get down to 0 ms eventually, but I'm not sure if they've hit that yet.
Similar techniques are used in video compression - encoding the exact value of every pixel is expensive, but you can trade bandwidth for processing by encoding transformations of a previous frame. A modern compressed video consists mainly of these interpolated frames, with only a minority of frames containing a full image. This interpolation can use data from both past and future frames (B frames) but can also use just the data in previous frames (P frames). This works extremely well most of the time, but there are some edge cases:
Mice and head trackers can already run at 1000 Hz. It's the GPU that cannot keep up.
Instead of black-box interpolators (e.g. Sony MotionFlow), a smart interpolator can be made to know the high-frequency controller inputs in realtime, and doesn't even need to use guesswork-based interpolation for everything.
Just shift everything around based on the high-refresh 1000Hz controller input. (In other words, "reprojection").
Also, knowing more data about the source (e.g. near-zero-lag controller input stream) eliminates lots of interpolation guesswork. It's much like how H.264 (video compression) is heavily interpolation-based mathematics during the video codec, but it had full awareness of the source video material, to successfully compress it virtually artifact-free.
So basically, you are simply giving a smart interpolator full awareness of things like geometry & input at a higher rate than the GPU renders. To avoid guesswork on those kinds of items.
Things like future multilayer Z-buffers can help solve a lot of parallax-reveal problems of trying to create intermediate frames, and there are future tweaks they are working on to eliminate reprojection artifacts. Like artifacts or reprojection distortions around edges of objects in front of objects. So adding intermediate frames with full parallax effects can eventually become artifact free because of the GPU's knowledge-in-advance of what-behind-what. Basically, more advanced reprojection algorithms that can create near-flawless intermediate GPU frames (without lookforward) without a full polygonal rerender.
Prediction helps (as it does for Oculus), but remember, we have controllers that already go at ultra high frequencies, and it is expected headtrackers will eventually become ultra high frequency too -- and that extra data can reduce the need to do lookforward prediction.
It's all very complex, with many researchers working on multiple solutions, but it can reduce the average processing-power-required per extra frame, and it can theoretically allow high reprojection ratios without lookforward lag (e.g. theoretical future 10:1, such as multiplying 100fps to 1000fps, at least with 1000Hz input devices like 1000Hz gaming mice, and 1000Hz head trackers).
Several VR scientists have indeed advocated the need for 1000Hz eventually, someday in humankind, as there are confirmed tangible immersion benefits to getting that high and beyond.
That's why I wrote that article full of motion demos explaining the visual science concepts of why 1000Hz displays are needed. It will be useful for passing a theoretical future Holodeck Turing Test (not telling apart a VR headset versus transparent ski goggles in a reality-versus-VR blind test), in terms of Morarity-style or Matrix-style "it's real" VR.
Many tricks layers upon each other, to achieve what's being achieved today, and this creativity will only continue. Lagless lookbehind-only interpolation (utilizing ultra-high-Hz controller input to reproject new 3D position). Foveated rendering too, yes. Realtime beamtracing with realtime denoising (NVIDIA scientist paper), perhaps. Maybe even all piled on top of each other simultaneously, perhaps.
So if that delay is 200ms, we know it makes people sick, for example. A delay of 0ms would be "zero lag" IMO.
Its quite possible that with progress made in brains wavelengths readings, the true vr in form of iphone1 will be a head cap you put on and your view/smell/touch perceptions are overwritten by cap’s sensor. That would trully be regular vr/phone versus true vr/smartphone/iphone1
That sort of headset would be such a massive breakthrough in neuroscience that the VR aspect of it would be tiny in comparison.
Inside out tracking is a reality in consumer devices now. All Windows MR devices that shipped late last year have 6DOF inside out tracking via cameras on the front of the headset with no external sensors. Moving forward there will be more devices from other vendors that use inside out tracking. Qualcomm has shown prototypes, Google + HTC were working on a tango device that got cancelled, HTC is working on an inside out standalone for the Chinese market, Oculus has shown standalone inside out tracked prototypes, etc.
They are very, very good. I've owned every major HMD since the DK2 came out in 2014, and I would say the Samsung Odyssey is the best one to date. The inside-out tracking is fantastic and just as good as Lighthouse (in practical usage, not theoretically). When you consider that there is no setup involved, it makes it a no brainer that this is the way forward.
Maybe they could do inside-out tracking on the controllers?
Agreed, there is a bit of an occlusion issue when your arms are behind/above the HMD. I feel like they could probably solve this though, with inductive tracking like Sixsense [0] integrated into the controllers and used in conjunction with IMU/camera data.
Also, in a perfect world this will all just be a transition step before we get full on holodecks.
We're doing our own small part to try and make that happen with our automated 3D scanning platform (http://realityzero.one)
If it’s a cartoonish game, the imaging doesn’t need to be photo-realistic. If it’s ‘toon style or lit and textured by photography of real environments, the polygon count doesn’t need to be high.
What it does need, beyond the hardware, is immersive physics, and a gameplay-justified reason for why you can’t run through the furniture/wall/cable that you can no longer see. The game “I Expect You to Die” does that perfectly because you’re sitting down the whole time.
As we finally realize, how not very real time our OS and software / hardware. After all these years of abstraction and slight delay added to everywhere in stack. We finally have a motive to unwind / improve those.
May be some day we could have Sword Art Online Style Game to move VR forward. Link Start.
Alone for remodeling my flat (kitchen, bathroom) or building a house.
I would also like to train on a virtual lathe before using a real one.
I might buy the new htc vive and i will see it as an early adopter beta hype thing because there is still work to do, but in general it feels already really good.
That Valve Portal Demo, wow that frightned me a little bit :)
You could immediately do a number of useful (read:productivity enhancing) things with the iPhone 1 - play your music, make calls, browse the web, take pictures, send emails. It deprecated a lot of what needed to be done on traditional phones, desktops, and laptop computers. It would last a day without charging, you wouldn't have to hook it up to a GPU or strap it to your body to use it, and it wouldn't give you motion sickness. Cellular technologies developed quickly to support the bandwidth needed for even better user experiences. Shipments jumped from 1M in 2007 to 20M in 2009.
There are very few polished games or apps available for VR 2 years after the "new" generation of VR headsets was released in 2016 by Oculus and HTC, and total headset unit shipments for the entire market (excluding phone-mounting headsets such as Gear) are probably in the low single digit millions for 2017. It hasn't yet deprecated any traditional dedicated communications technology or functions.
I wish I could get excited about the future in this field, but I really just don't see what the killer app will be for VR. Facebook thinks it will be virtual meetings for the enterprise and hanging out virtually with friends/family for the consumer market...I am very skeptical but want to be proven wrong as VR is one of the last platforms pushing hardware and software innovation forward at the moment.
Not true. Examples: Brass Tactics, Robo Recall, In Death, Lone Echo. There are already more quality games in the Oculus Store and Steam than most people will have the time to play.
Previous VR endeavours, like 90s, early 2000s would then be the Pac-Man and Donkey Kong in this analogy :-)
We're most definitely there. Here's the Original DOOM, modded for VR:
As someone who has had an oculus since the consumer version was released, my main problems are:
1) Eye strain. Even though you have a 3d effect you're still looking at something a few inches from your eye and that disconnect causes eye pain and headaches after playing for more than an hour or so.
2) Locomotion. I've yet to find any way of moving around in VR space that doesn't either make you nauseous or pull you right out of the realism of the experience.
Locomotion is less of a problem than people who think its a problem is, and I think that stems from a lot of people in VR being hardcore gamers, and exploration of large spaces being a core mechanic and selling point for 3D video games for the last 25 years. I don't think omni-directional treadmills or vestibular stimulation or anything inconvenient like that will catch on for locomotion, and I think we'll either use various forms of teleport, or sliding (traditional 3D) locomotion for the foreseeable future, and people will mostly be ok with it. It's possible that with wireless and/or standalone headsets that redirected walking, will be a popular option. You could imagine a headset where Chaperone\Guardian builds on SLAM used for inside out tracking, and can give apps information about the layout of your house, allowing for large procedurally created virtual spaces. This still doesn't solve the problem for people that don't have medium sized private spaces to play in, and also makes it harder to do multiplayer games where players are playing in very different spaces.
And those things were true.
Games and people adapted.
Personally, I feel something work that can pick up nerve impulses as well as deliver modest feedback, tactile and or electrical will close much of this gap.
New input paradigms will advance too, just as they currently are for touch.
Touch today is getting good. The finger in the way problem is being chipped away.
VR was already too expensive. Now it’s completely out of the question for most people. Adding an absurdly high res and high refresh HMD to the mix right now doesn’t seem like a good idea.
There's going to be a black bar across the top middle of my field of vision?
We already have 3d video - I'd be surprised if the concept couldn't be expanded, leaving the unit to merge streams of background environment and 3d animated sprites.
High res stereo at 120Hz is never going to have to same graphics as the latest high budget big game release. Current GPUs are already very powerful, but if people have the expectations of getting the same graphics when they use VR they are going to be very frustrated.
At some point your GPU runs out of things to do.
The reality is that greater horsepower allows for greater abstraction, and easier to program APIs. Increasing developer productivity 2x reduces performance 10-100x, or something like that. So there’s never “enough” performance for the same reason there’s never “enough” powerful/usable APIs.
Audio used to be really difficult to process in real-time but now it's trivial. There's only so much audio processing you can do before it's ridiculous and pointless.
The same goes for video. Once you have, say, a 40K display for each eye at 244Hz there's no point in going for more pixels or faster refresh rates. If a GPU can handle that, easily, then that GPU will probably be best put to use doing other things in addition to rendering graphics.
Memory is not tied to your senses, we can always find uses for more. Audio and video are, and at some point it's as good as real.
There are a few things that need to be accomplished before widespread adoption:
- Removal of wires. It restricts movement too much and removes immersion. The new HTC headset is a step toward this.
- Higher resolution screens. VR AMOLEDs like this are a step in the right direction.
- Prices for GPUs need to go down, and/or a few more years are needed for average computers to be able to render high frame-rates without breaking the bank.
- Headsets need to be lighter and smaller.
- Removal of sensor placement the room. This will be harder to do, but cameras/sensors built on the headsets themselves could potentially accomplish this.
The way I see it, we're in the iPhone 1 stage of VR right now. Imagine the iPhone X version: lighter, smaller, higher resolution, more colors, higher frame-rate, less hassle. These are all inevitabilities, and at that point it will become much easier to adopt the technology. We're also missing a true "killer app" that will get people to purchase a headset JUST for that. I think it will take some sort of truly massive MMO the likes of WoW to accomplish that.
The future is definitely exciting in this field. I hope hardware vendors don't give up and can see the light at the end of the tunnel.