The distances in the milky way are mind boggling. But the age of the Earth is even more so.
Now imagine a people as far advanced of us, as that ant is to you. Now imagine that 10,000 times more advanced. Or 100 million times more. That is the kind of difference we have to consider.
The unfortunate reality may be that other civilizations are so far advanced from us (100x? 10,000x? 10 billion?) that we are effectively 'alone' all the same. Like an ant pondering the garden hoses, BBQs, or fence posts of the universe.
During that time, the population of Europe increased as well - many people just thought that America was a better place to live.
There's no need to wait for "full saturation of a star system" to send out probes and colony ships.
I made the point in another comment but I do think there has to be a reason to send out colony ships. I agree that exploratory, unmanned probes would be sent out to gather information but I doubt sustained efforts to colonize new star systems would be made until there was significant social, political, or environmental pressure to do so. One ship colonies never succeed beyond a toe hold, you need a constant flow of people, resources and information to drive the founding of a new civilization especially if the trip is one way with little possibility of trade. If all you needed for the development of a new culture and society was an initial seed population and time then every town in the world should be a New York.