The counter is that most of the usable mass in the universe is in interstellar objects such as comets and free planets which are 50% water ice or so, rich in organic materials and can be consumed completely to build habitats.

With D + D fusion (easy to believe compared to CNO) an interstellar civilization could be essentially independent of stars.

I'm sure part of the resolution of the Fermi paradox is that dry inside-the-frost-line planets like Mercury, Venus, Earth and Mars are not the generic places you find life.

If things were a bit different I could see a would-be Galileo on an less cooked version of Io or a more cooked version of Europa who looks at the Earth with a spectroscope and sees oxygen and has a huge amount of trouble with his local "church" that thinks it is impossible to have life on a planet with less tidal activity, less radiation, inside-the-frost-line dryness, etc.

Even if creatures like us became interstellar we might have millions of kilometers of condominiums and shopping malls in the oort cloud and could care less about inner solar system planets.