The poor UX of moving the user camera in the virtual scene is trivially avoided.

If any of the immersive apps drop framerate, you'll feel it. Its not just a games concern.