I have experienced what you're describing while gaming on my personal VR headset, but never with Vision Pro.

I'm not sure if that's due to the passthrough or the fact that I was only ever using it to actively do work (and I usually mean work on stuff for the Vision Pro itself, not just "normal" dev work).