It's not a matter of Earth becoming uninhabitable. You're right that it would take a lot to do that, though it is possible with a large enough asteroid or some unexpectedly fast acting runaway greenhouse effect. The problem is of human civilization continuing at its current level and continuing to advance. Any number of small disasters could set us back hundreds if not thousands of years. We have direct evidence and record of this happening multiple times throughout human history so we know it can and probably will happen. That's why we need a self-sustaining society elsewhere, preferably far enough away to resist whatever disaster causes Earth to fall.