You need everyone to have VR gear at a ubiquitous level similar to monitors and I just don't see that happening.
AR is a much easier sell with a much lower cost to entry
It's also true that while driving on a highway you are putting your life in the hands of other drivers, some of which may be drunk or tired.
When you fly, you are at risk of the plane crashing and you have absolutely no recourse.
I can absolutely assure you there are some people who would use VR in public today and some who will refuse to ever do so, but let's not pretend that closing your eyes and ears off presents some fundamental transformation that is 10000x more dangerous than any experience humans have ever voluntarily subjected themselves to.
I'm sure you could find someone in the 1600s who would make the same claim about wearing noise canceling headphones if someone from the future were to explain the idea to them.
This means most public places are just not going to work. Even in your own home, you might be unwilling to disengage so completely if you have a mischievous sibling, an animal who might get underfoot or delicate/dangerous items near you that could be hit by a wayward swing of the arm. In general, you are completely reliant on others around you to warn you of danger - from wandering too close to walls or furniture, to watching out for others who might want to take advantage of your vulnerability.
Offices might work, but not open plan. Classrooms are a non-starter. Airplane seats would be good (if not great), but not public transport like a subway or a city bus, even if you disregard the nausea issue. Even sitting in a high-end cafe or a public library, you'd need to have someone actively looking out for you. Again, would you be comfortable laying down for a nap in the middle of a park? If not, then using VR there isn't going to work.
This is an unavoidable issue that can only be slightly mitigated with automated detection systems. There are inherent limits to a technical solution like that: Someone staring at you intently from a short distance away to gauge how they could best run up and grab your bag isn't something a proximity warning system would be able to counter. Already just concentrating on portable video games in public (phone, Switch, etc.) is such a distraction that you have to be wary of your surroundings - the idea of adding in complete sensory deprivation to it as well is farcical.
VR simply isn't the next smartphone in terms of technology. It will forever be a niche used at home, alone. That isn't to say this is a small market - the entire video game industry has a similar use case - but Zuckerberg has his hopes set on a revolution akin to the iPhone and it just isn't going to happen.
AR is a much more realistic opportunity. If Zuck was as smart as he thinks he is, he'd pivot and focus there. But I don't think it will happen until Apple launches their solution (whatever it may be), and by then it may be way too late.