I see your point. A retreat need not always be in far off places or for 1 week. Infact, I think, most of the startup teams can achieve what they want in 2 or 3 days max.

In our case, there are team members who can't attend the retreat and we are fine with it. There is no evaluation/judgment happening. But we encourage them to be there to learn and understand faster