You don't need to hand wave physics away. von Neumann probes + a lot of time would give space faring civilizations ways to arrive at the furthest reaches of the galaxy, even with our current understanding of physics.