Nope, Nokia 3230 (Symbian 7.0) was first with such functions - it has exclusive AR-game «Agent V» with depth sensing capabilities.[0]
It's entirely plausible that AR ends up being a dead end. It's 5 years later and still not really a big market.
Headsets, I think the current devices are just a preview. I'm skeptical that transparent displays are strictly necessary for AR headsets, and I have seen some industry speculation that the HoloLens 2 is probably at the limit of what can be done with waveguide displays. Unfortunately, waveguide displays are just about the best sort of transparent display tech available for this particular use case, so we need to either shift to pass-through-camera displays or invent a whole new display technology.
I'm betting more on pass-through. I think the tech is already available, what is at issue is decades of software cruft in modern operating systems making low latency through put too difficult to achieve.
I am guessing there are uses for it in industry, although I'm not sure where it would find the best product-market fit.
A futuristic AR device that would work seamlessly would truly "disappear" into the environment. No HUD necessarily (at least in terms of what Google Glass was trying to accomplish) but imagine if people, places and things could register actions in the real world, sort of like "Press X to open door" in a video game.
Imagine a sort of DNS-type registry for registering ownership of physical space tied to geospatial mapping data. When you walk into such a space, your AR device could unobtrusively indicate that there is something interactive where you are, and who owns it/what it is.
Perhaps your city owns public basketball courts. They could register a virtual fence around it and when you're on the court you're presented with the option to start basketball stats tracking and a tutorial with a pro from the city's NBA team.
Or something more commercial could be: you walk into Starbucks and your drink is ordered for you automatically and you can see the estimated wait time.
These aren't some sort of experience designed for entertainment value, they're truly small improvements which augment your life. Your choice whether or not you think such a thing would be necessary or good for you, but a lot of it already exists with phones. AR used this way would simply make phones obsolete.
Obviously, the processing power wasn't up to the task. It was a bit like trying to go back to an 800x600 CRT for work. But the HoloLens is a 3.5 year old AR headset by now. What I did get out of it was a glimpse into what "could" be, for some value of "AR headset".
First of all, I wasn't tied to a table. These days I don't have a desktop anymore, but even my laptop is basically unusable without a table, especially for the work I do, where I basically require the use of a mouse.
Second, I wasn't limited to my one display. I could put windows anywhere. If you take the virtual desktop concept, and throw it away, scoffing at is as a child's toy, you might get the idea. I kept Twitter hanging out on the front of my refrigerator. I only checked it when I got up to take a drink. I kept my email sitting over to the right of me. It was never directly in my field of view, close enough to be convenient to check, and never hidden behind other windows or minimized away to require hunting for it in an ALT+Tab view.
Third, the system remembered the location of all of my apps between sessions. They became more like appliances in my mind than windows on a screen. The physicality of it, and the recall of it, surprisingly made it more enjoyable to use.
Fourth, the voice assistant was significantly easier to use, now that it was running from a microphone sitting right next to my face, rather than one sitting right next to my GPU fans. I found I started voice commands for basic tasks like opening and closing different applications.
Finally, when I was done, I could just take it off. There is no omnipresent display to wake up and show you a notification icon. There's no need to keep a dedicated space for it. I started viewing my desktop like a garden plow: a big piece of equipment that needed space (which was certainly at a premium for me at that time) and required a lot of manhandling to use well.
Application-wise, I think it's hard to really describe what is "good" about AR. Either you talk about existing apps, in which case people say "yeah, but I can do that on my computer now". Or you talk about potential apps, which is too abstract for people. Try going back to the early 80s and explain to an MS-DOS power user that they'll be using exclusively GUI systems in less than 20 years.
Because of the way it blends into life, I think AR has the potential to improve the overall computing experience for anyone who already uses computers. The potential is to replace your laptop AND your smartphone, not to be a device in addition to them.
That's what John Carmack now says is the argument for VR. It's not about virtual reality. It's about virtual screens. Now, you can have a wide-screen theater experience even if you're in some tiny apartment in Guangdong or San Francisco. And hang out with your friends, even. Think Sunday Night Football, not Doom VR.
VR requires applications that can't be done on flat-screens, just as movies require content that can't be done in books and radio requires content that can't be done on MP3 downloads and TV requires content that can't be done in a movie theater.
For example, apps like Tilt Brush can't be done on a flat-screen. Yes, you can create 3D content on a flat-screen, but Tilt Brush is not just a 3D content generation tool. It's more of a 3D painting tool, a creative release rather than a productivity tool, and it's a social network of content from other people that you can explore. On a flat-screen, it just wouldn't have the same usability or impact.
There is a VR Digital Audio Workstation-like app called SoundStage that--while doesn't have the usability of Tilt Brush--does signal some interesting concepts for a potential Tilt-Brush-like DAW.
I originally didn't think Google Earth in VR was a good use of VR, but since starting to work in the language training industry, it has been a surprising source of content for us to improve language training for adults. As an app on its own, I'm not sure it makes a whole lot of sense past a basic demo, but the content and form factor are very useful if one can apply a purpose to it other than just exploration. (Though the exploration can be fun for a short time, too).
I think the use of action-oriented VR titles for exercise cannot be understated. It's pretty much the only form of exercise I have any patience for, anymore (between children, job, volunteer work, side projects, high-cost-of-living in the DC metro area, and a sports injury or two). Obesity is an epidemic. Mass adoption of VR could fix that.
As an addendum to that previous point, I also really enjoy VR meditation apps.
VR could (and in some offices, is) revolutionize teleconferencing. I've had meetings in AltSpace VR that, even 5 years ago, felt better than using Skype for remote meetings.
Right now, we have a very high cliff to get to be able to create content for VR. I think it's a very similar issue to when GUI operating systems first started to come out. If all you knew were CLIs, it was very hard to see the utility of the GUI. And when people first started adopting them, they frequently had to dump out to the CLI to do things. Less than 20 years later, the CLI was relegated to niche use. I don't think the vast majority of people could have anticipated the shear number of apps we use, the mass of files we manage, the ubiquity of computing in our lives, in the mid 1980s. But all of those systems that make that possible weren't a part of those first GUIs.
Most of the examples you give though emphasize novelty over utility to me. I agree the Tilt Brush is neat and I think that absolutely has an application, but I think it's similar to MS Paint. MS Paint was a revolutionary program iconic with Windows, but it was a feature attached to the actual product of Windows. Not many people bought Windows just to use MS Paint.
I am still failing to see the actual product case for VR, outside of niche circumstances. Most early adopters I know in the gaming space have largely abandoned it. So far, from an adoption perspective, VR seems like a less successful device than the MS Kinect. The Kinect wildly outsold VR headsets (talking an order of magnitude difference) in the beginning, but then people just sorta stopped using it because they felt it was largely a gimmick.
Most of the stuff you mention is all about potential, and it's wonderful to have believers such as yourself because you are the guys that push to create an industry. But right now VR is still in its infant stages, and personally I feel like it's been around long enough that we'd have some idea if it were going to take off.
I really do hope you're right and I'm wrong and there is a really cool technological medium being developed that will have a great impact on most (or even many) people's lives...but I don't currently believe it.
It's been there for decades. I tried Jaron Lanier's original VR rig in the 1980s. Autodesk's early one around 1990. W Industries somewhere in there. HTC Vibe and Microsoft HoloLens last year. It's fun for a while, but not compelling for long periods.
Except for Beat Saber. That uses VR right. You get to move your body, but you stay locked to the real world reference frame, so it doesn't induce nausea.
Valve promised to release 3 big titles by the end of the year, and it's November with no news yet.
https://www.techradar.com/news/half-life-vr-game-could-land-...
The problem I see is that VR adoption is so low there's little incentive for developers to target for it since the development is so expensive. And this in turn means fewer people are excited to buy VR - it's the chicken and egg problem. Facebook was supposed to solve this by throwing gobs of money at studios, but considering how much money they spent that was an absolute failure.
The reason VR isn't taking off is because most people don't find VR worthwhile. What it would take to make it more worthwhile - better technology, better price, better content. I think the tech is 2-3 years away from being good enough for even early adopters. So far people that use VR are mostly hackers, developers, tinkerers - not many of them are really consumers first. I think the price is 5-6 years away from allowing it to transition to mainstream. But the content? Honestly this was what I most excited about years ago, but now I'm beginning to think VR is just a dead end. Yes, there's some cool stuff - but very little of it isn't a gimmick.
Did you miss the recent Five Tilt Kikstarter?[1] It is a new display technology, that I'm eager to try next summer if they manage to deliver. It needs a reflective screen somewhere in the room, so it won't work as mobile; but for anything room based (games, teaching and office stuff) it gets a lot of bullent points right, and I think it could replace a lot of use cases for VR as well.
[1] https://www.kickstarter.com/projects/tiltfive/holographic-ta...
What kind of elitist nonsense is this? The reason AR isn't taking off isn't because the users aren't good enough for AR, it's because AR isn't good enough for the users.
> It takes time for new technologies to become widely adopted and utilized.
My point is some technologies just end up not being very useful, ever.
Could you expand on this, please? I didn't realise this was a significant concern until now.
Apple on the other hand focused on AR that works with normal phone cameras, allowing all their devices to get ARKit support, which is what Google ended up doing too. They dropped Tango for ARCore, which runs on any device with a normal camera on it.
Specialized hardware will always lose, because it's stuck in the content -> hardware -> content vicious cycle. No one buys it because there's no content, and no one makes content because no one has the device.
And it doesn't require those weird-looking and weird-feeling controllers to interact with the world.
So... maybe VR is not yet there, because the hardware does not allow for the full immersion that we unconsciously expect/want from a virtual reality?
You can either keep your experience confined to the user's play boundary. You can teleport people to new locations. You can let people walk to the edge of their boundary and use a gesture of some kind (e.g. flicking one of their thumbsticks on their controller) to turn them around to do "redirected walking". You can provide a "strong stomach" mode to walk at constant velocity in the direction the person is looking. Or you can stick the user in a "vehicle" with clear visual cues that they are in said vehicle. And that's it. There is nothing else you can do.
For example, Doom VFR only has teleportation movement and it results in an experience that doesn't feel very Doomlike. Instead of blitzing around like a rabbit with a gun, now Doomguy is a magical genie with a gun.
To start, I think it's a mistake to approach VR application design as "this app, but now VR and immersive", whatever the app may be. It might get better in the future, but for now there are too many compromises for the user to start your design process in that way. VR requires a bottom-up rethink of what you're creating. If all you are creating is "X but in VR", then you're not actually adding any value for the user, while simultaneously asking them to commit to the economic, ergonomic, and context-switching costs of entering into VR.
I am extremely skeptical that the first-person-shooter will ever be successfully ported to VR in such a way that it will continue to be of the same mainstream interest that it currently has. I see it as a similar issue to how the Adventure genre went from being the one of the most popular to an extreme niche with the advent of 3D graphics.
I think FPSes are popular on flat-screens because they specifically aren't realistic. Real combat is the most grueling thing one can do, basically by definition, because if you don't try harder than the other guy, you die, so you have to try your hardest and hope it's enough. The mouse-and-keyboard or gamepad system with a stationary screen completely removes that issue. And while I enjoy FPSes on flat-screens, I've personally found the experience of simulating killing people in real first person view to be psychologically disturbing.
I play FPSes strictly single-player, as I can't stand the other people invariably get paired with in public multiplayer, which I think we all know is a very common problem. If VR could kill off the FPS genre and cut out a huge swath of the most problematic portion of the gaming "community" (this issue is why I always weasel quote the word "community" when referring to gaming), then I think we'd all be the better for it. You don't see a lot of GamerGater's complaining about "SJWs" in the context of strategy game releases.
So there are very significant technical and ethical reasons why I don't think we should even try to be porting FPSes to VR, and instead focus on revisiting old genres and discovering new ones.
There is quite a few popular milsim FPSes which kind of favor realism (to a reasonable extent), both in in-game physics (weapon, vehicle and player body behavior simulation) and visual depiction.
I think FPS is popular because this is one of the fundamental genres out there: a depiction of a combat, as perceived by a combatant. This... err... trope has been there since the dawn of men, people imagining themselves fighting. The advent of 3D graphics brought that to computer gaming.
Adventure is another fundamental genre, and I'd disagree that it's niche. See below.
> [...] basically by definition, because if you don't try harder than the other guy, you die, so you have to try your hardest and hope it's enough
I'm sorry but I just don't understand this argument. This applies to every competitive game out there, no matter the genre. If game has this concept when the best performing player wins and other player(s) lose - you have to do your best.
> I've personally found the experience of simulating killing people in real first person view to be psychologically disturbing.
That's a problem unrelated to VR. Although it's somehow aggravated by it, as VR strives for more immersive experience, thus, more realistic depictions of in-game events.
I wonder if you should compare this to airsoft games. They provide a realistic immersive environment (in my perception, immersiveness the primary promise of VR), and they're sort of FPS-in-real-world stuff.
> stand the other people invariably get paired with in public multiplayer
First, I believe this has nothing to do with game presentation layer (like VR or even 3D at all) or genre. Second, there is an awesome solution for this: playing with friends.
As for the genre - I assure you, you can find less than enjoyable opponents in any game genre. Especially strategy games (MOBA/ARTS subgenre, specifically, hehe), but not limited to that.
> If VR could kill off the FPS genre
I don't think that's possible. Again, the advent of 3D graphics brought the genre to computer games, and you can't put the genie back in the bottle. VR - if it becomes mainstream - could transform the genre, but can't kill it. The fundamental idea of a simulated combat as perceived by a combatant is way too high-level concept.
Also, I'd disagree that your "adventure used to be one of the most popular and now it's niche" statement. Adventure genre concepts are still as popular as they always were. Storytelling, exploration, and puzzle solving are key elements of a lot of mainstream games out there. My understanding of what happened is the genre fusion, not a death. A specific UI design patterns (esp. those of point-and-click adventure games) became less popular, but I still wouldn't say they're gone "extremely niche".
> and cut out a huge swath of the most problematic portion of the gaming "community"
I'm sorry but this sounds really weird to me. My understanding is that our world doesn't work this way, so your idea feels strange. It's not like $genre games make people toxic, and if it's gone then those people just vanish off the face of the Earth. If the FPS genre somehow magically disappears, the desire to interact with others would lead them to other pastures.
There's rampant speculation that Apple will deliver a glasses experience where Google already failed. I think this is possibly the last chance for AR to get traction before going back to the shelf for another 20 years.
In other words, moving from VR to AR would require a ton of extra work to integrate the real world. But moving from AR to VR should be much more straightforward, albeit in many ways equally challenging because now you have to worry about two worlds - the normal overlay of AR and the environment from VR.
Now that I’ve said that, maybe I should think of them as more like related disciplines instead of special cases of each other...
I'm not sure what's going to happen first - technology to do proper AR like this, or fast enough, high-resolution (both spatial and color) enough cameras and displays being available to do the "AR via VR" option.
VR is all about total immersion - sights, sounds, movements and everything else. The entire world needs to be contained in the headset, and what's happening outside it is irrelevant.
AR, on the other hand, is about interacting with your surroundings. Immersion is cut down, but the system now has to be intelligent about the outside world - recognizing objects, projecting at the right places etc.
Really? Why not?
First of all, Apple's ARKit (and by extension, Google's ARCore) is not a good experience. It has some "gee whiz" factor to it, and then once you get over it, you realize that there really aren't very many things you can do with it that you can't do better with a smartphone or a VR headset.
Case in point: I spent a lot of time at my previous employer working on self-service/guided-repair AR apps for the iPad Pro. It's a ridiculously bad concept (though my previous employer is kind of known for chasing ridiculously bad concepts). 1) Your hands are occupied with the tablet, so you're not actually able to perform the repair tasks without putting the tablet down, thus largely negating the usefulness of the overlaid 3D animations we created, and 2) who is going to take a $1000 Apple product with them out to a repair job?
We also spent a lot of time making large posters into 2D animations for conference expos. You could have had the same, exact experience for 1% of the cost if you just bought gigantic TVs and turned them to portrait view. I mean, I didn't complain about getting the money, per se, but it just was not good stewardship for our clients to even propose that contract from the sales team.
There are distance measuring apps that presume the tracking is accurate and precise, which it is not. For a very long time, ARKit had very little concept of how far away any given surface was. Instead, it was good about scaling objects so that visually they did not appear to be moving closer to you as the tracking point jittered wildly on the Z-Axis. At best, it's better than walking off steps to get rough measurements of a space, but anything more than that and you're going to have an easier, faster, more reliable time with a $10 retractable tape measure.
Second, the HoloLens is an amazing device, but it's nowhere near ready for any mainstream use. I think it does have a good use case in industrial repair scenarios (which was originally how we got the iPad work, I had built an app for HoloLens that did the same thing, but then the suits got involved and wanted to push our Apple partnership). There are specific features of the HoloLens that make this work, and there are specific features of the job that make the HoloLens work. Specifically, HoloLens' built-in, completely free voice recognition is a huge boon for being able to build systems that you can use while your hands are occupied with the job. Second, having physical objects (the machine you're trying to repair) on which to hang virtual graphics causes a huge improvement in believability of the visual and usability of the application (primarily related to issues of object permanence being hindered by the limited field of view).
That last point wouldn't be so bad if the HoloLens were visually and performance-wise up to spec with much, much cheaper VR headsets in terms of grpahics and visual fidelity. But so long as you're paying literally 10x as much for half to a quarter as good of visuals (not to mention the input systems being otherwise mismatched), you have a lot of ground to catch up on to make an AR-only experience as valuable and meaningful to a user as you could a VR experience.
AR will continue to be a lackluster consumer user experience for a long time. Most developers making apps for these devices don't understand how to make a real AR app. In AR, the user supplies the context to the developer. In VR, the developer supplies the context to the user. This fundamental dichotomy makes good, meaningful AR application design significantly harder than VR design. In my own evaluation, easily 80% of the extant software in the HoloLens and Magic Leap stores take no cues from the user's immediate vicinity, in their context, to which they are tying their experience. In essence, they are just crappy VR experiences.
In comparison: VR is cheap, it works, it is relatively easy to design for, and there are already a ton of great apps available for it.