I can imagine that. My personal take is that people who have a good sense of balance can't handle VR when the player moves. Games like Beat Saber get around this by making it feel like the objects are coming at you, rather than you at them, but when you're physically moving around in the world (and especially in non-standard ways like would be the case in a Portal game) people start to feel sick while playing.

Maybe it's just a matter of getting used to it?

I would looove to try moving VR games with an omni-treadmill some time!