Giving shits about things is one of the big reasons we have wars and other potentially civilization-ending possibilities.
Even if we are able to put 100 humans on a spacecraft for 100 years hurtling across the galaxy I'm pretty sure they'll start fighting among each other after 2 years. It's one of the reasons that I think we'll need to make embodied AGIs that are less prone to fighting to continue the legacy of civilization long term.
If there is no expansionist species, after a time one will emerge due to mutation over time.
Look at humans - there were a bunch of other intelligent species around the time humans came to the scene - we out expanded and outcompeted all them.
Now of course this is just a sample of 1. But also seems kind of inevitable. So if mutation and natural selection are the only way for an intelligent life to emerge, it would probably also be expansionist, just by virtue of outcompeting everything else. And thus would naturally want to “explore” for the sake of it - just like humans.
Of course there might be other ways to achieve intelligence- like it being developed by other species (AI). But same rule still apply. If there are more than one, eventually the one that wants to expand will expand more than the others that don’t…
And even if the species found a way to “change themselves” to make them not give a fuck, by a virtue of change being possible eventually you’ll get expansionist “strain” again.
For such a species, there would likely be no selection pressure towards such a trait. I suspect it's actually the opposite: being less expansionist could have enough benefits that it would become widespread entirely through random mutations.
Infinite growth isn't possible, desirable, or even "natural".
For that reason, only warp drive harnessing star-equivalent energy levels would be visible at distance of stars and so-on. I mean, Fermi's observation comes down to "there's no intelligent-being signature at the star-levels of energy we can see". But that could be because: intelligent beings are rare or because harnessing star-levels of energy is hard or impossible no matter how advanced a society gets or advanced societies have no need for such harnessing or because advanced societies follow the paranoid "dark forest" logic.