More generally, presence is the missing piece. VR is one possible solution to that, if you can solve the motion-sickness issues. Holographics (if real-time holos ever reach commercial viability...there's some interesting research on it now) is another possible solution. Perhaps there are other possibilities that people haven't found yet. Wrap around screens, like the Google Liquid Galaxy display? Projectors?

But someone needs to invent a way for office workers to feel like they can have an impromptu conversation, casually, with a coworker thousands of miles away before remote work really takes off.

To me, VR is presence. The metaverse has existed since the first network was turned on. It's the sensation of presence in the metaverse that people are referring to when they say "VR".