I used to think this until someone pointed out that the earth isn't a fixed distance from the Sun.. it varies throughout the orbit from a minimum of 91 million miles to a maximum of 94.5 million miles away.

So there's already a 3.5 million mile margin of error in our Goldilocks zone... we'd have to go way beyond a few miles of annual change to feel any impact.