Looking at a far object relaxes the muscles, while looking close tenses them. Any deviation from this is somewhat uncomfortable. Fixed focus displays mean that, when you look far, your eyes relax, expecting the physics to be correct, but then have iterate to focus at 3m (or whatever the fixed distance is). Same with near objects. The eyes tense, things go wrong, then they drift back to 3m.

Blurring can be applied, but that doesn't change the physics that your eyes expects, which causes the disconnect and discomfort, which varifocal displays solve.