I'd rather follow a set of virtual arrows to my destination than have to keep checking my phone. This goes for a lot of tasks I use my phone for. Though the privacy concerns are more worrisome than mobile phones!
VR isn't going to be a novelty if people switch to experiencing life primarily in a virtual manner due to the vast expanse it will offer. That includes travel, art, gaming, social, shopping, media.
With today's technology? No. With the technology we'll have in 30 years? Yep. And particularly the technology that will gradually evolve over that time. It'll be far more impressive and useful than the smartphones we have today, in every regard.
If you had told someone in 2003 that billions of people would use their cell phones for so much today, and that people would practically be attached at the hip 24/7 to them, I don't think you'd have been believed - the notion that everyone would have their face buried in large glass screens for hours each day, socializing, shopping, etc.
The real threats for Apple, imho, are successful web companies expanding towards hardware. Apple does great HW and poor online services; someone doing great online services and great HW could hit them badly. Google got close already; Microsoft will try again once they complete their retooling; Amazon are unfocused but could fish a winner with some original thinking; Facebook/Dropbox/others might also challenge at some point. Luckily for Apple, hardware is very difficult to get right and get profitable.