Saying "a question of time, not physical limitations" however is the handwaving of physical limitations "because technology will advance" thing I wrote about.
Why would it be a question of time?
How is time gonna make "faster than light" travel possible, or bring the closest stars any closer than many light years?
If anything time has shown that space travel capability can go worse and languish completely for decades just as easily as some external force (like the Space Race in the Cold War, or easy excess credit to fund things like SpaceX today) makes it so -- it's still been 40+ years since we last went to the moon, and the US had let languish the capability to launch people into ISS for decades...
Regarding technological development, again, on a long enough timeline, I don’t think it’s much of a concern. Technologies will continue to develop, quickly or slowly. It hasn’t even been a century since we walked on the moon. Deriving trends from this short period of time is not particularly relevant.
Between one unhabitable planet and another, plus some gas giants, etc., where's that plenty of space? Unless we also handwaving terraforming into the equation too (as we'll do with carrying any significant number of people via spaceships as opposed to small scientific missions, the 9+ months travel time, space radiation, lack of resources to eat/drink/breath there, etc).
Plenty of space in our own solar system
After all, what can you hide behind floating in space, fully illuminated by the sun?
or something in which you can travel and survive living in space (mobile space habitat).
How will you overcome the lack of gravity causing muscle/bone-density loss?
assuming you camp out near the sun and derive power from solar - how do you get food and oxygen? you'll have to assume extreme air, water recycling, and some way of producing food.
We’re also talking about hundreds or thousands of years here. It seems totally reasonable to me to assume that a private spaceship priced at ~$500,000 in 2021 dollars will exist by say, 2500.