Headphones are in a sense "additive": they take a channel (auditory) that for a prolonged period of time might be underused while we go on about our lives and utilize it in an unobtrusive way. You can listen to podcasts while doing other stuff. We can optionally upgrade the experience: If you want, you can close your eyes and lose yourself in music.
Goggles are necessarily subtractive. Whatever you do while immersed must justify the fact that you are obstructing your primary contact with the world. It may well be that there are worthwhile pursuits but its a bit odd that they don't advertise themselves.
Imagine using your phone but you don’t have to hold it up to your face, or tilt your head to look at it. You don’t need to sit at a desk, because your work screen is wherever you are.
If you really want the old experience, you could make the contacts emulate a fixed display in space, or on top of a soap bar.
The goggles today are much worse than the theoretically perfect version, but it isn’t clear where the experiential threshold is for “better”.
And that's coming from someone who spends a good amount of time doing studio work. I much prefer the comfort of a good set of speakers as they're far less intrusive, but for some things you just have to have a pair of headphones on hand.
This is why I have doubts that the current form of VR headsets is going to be the new standard for working.
We can make devices that are fixed in space, where you need to orient your sensory inputs around the device. Or, we can make devices that instead are themselves oriented around your sensory inputs. The latter tends to be better for ergonomics and convenience (at least), since the device is accommodating your senses instead of your senses accommodating the device.
In other words, AR/VR/etc devices are headphones for the eyes. I can see why it seems like an obvious next step to some (if they can get it right).