You could also ascribe the same issues for a parent who wanted to leave the house. In fact more so, since if I left the house with a small child, I can't instantly return if I become aware of some urgent situation. Yet, bars, pool halls, bowling alleys, movie theatres, restaurants, etc., etc., still seem to have survived this seemingly momenumental issue you are harping on about.
Couldn't you make the same argument against headphones in general? Those seem popular enough. I've even known people who used sleep masks and ear plugs when going to bed. Realistically, VR doesn't actually take you anywhere. You're still standing right there, and if a child or spouse needs your attention they can still raise their voice or even just grab you to get it. I don't think the immersiveness itself is much of a problem for most people, although there are certainly times and situations where you wouldn't want that because you need to be more aware of your surroundings.
It works perfectly well, to the extent that any single-player activity works with spouses and children. Source: I have a spouse, a child, and a VR headset.
Who does this work for? It doesn't for people with spouses and children. I can't be removed from the real world when my child wants my attention. How many hours of game play do you think the average family person might have available to them where they can just remove themselves from the real world?
Given all of VR's problems, the people who will play VR games seriously enough to spend money on AAA VR games will always be a smaller subset of the overall gaming population. It's never going to be bigger than what exists already. All these billions dumped into VR as the next big thing seem to miss that the VR market is limited by the problems the parent comment mentioned.