Distance is not such a great obstacle if you have time. Aliens could conceivably live much longer than humans, making many thousand year journeys feasible. Or they could send effectively immortal machines instead, maybe via something like von Neumann probes. Generation ships are also possible.

These distances only seem long to us because we're thinking mostly in getting to other stars in a single human lifetime, which is a pretty short lifetime even compared to some other organisms on Earth (like trees, not to mention organisms that can go in to stasis indefinitely, like the mushroom spores of Terrence McKenna alien contact hypothesis fame). Once you remove the limit of a single human lifetime, your reach expands greatly.

The many billions of years that the universe has existed should give an advanced civilization plenty of time to visit the entire galaxy, either in person or by proxy, without even having to invoke the possibility of FTL drives or wormholes.

If that's the case then we definitely do not want to face the universe as primitive indiginous people unceremoniously discovered by technologically mature explorers.

How did the settlers get to that island then?

Boats, obviously. But the settlers themselves were dead. Their descendants may have had oral traditions about how their ancestors reached the island but I don't think we have any evidence that they constituted a seafaring civilization. Those weren't really possible until the invention of reliable navigational instruments such as the compass.

The first settlers got there by a boat thrown off-course by a hurricane, the the seafaring skill was lost, so communication became impossible.

Fair enough.

That's not really relevant to the analogy.