My understanding is that you only have a relatively small area in the center of your vision where you get a clear image. And so your eye moves back and forth so rapidly that you can't tell without slow motion, sort of scanning in the environment into a 3d map.

So maybe some of the variability is in that mapping circuitry.