If it is indeed the FOV that makes the difference, this implies a potential solution for when a game needs to move the camera independently of your body: limit the FOV as the camera moves.