There is always someone huh. Such an arrangement literally sounds like the dream remote arrangement to me. It never even crossed my mind that there would be people who would say no to a fully paid pseudo-vacation every couple of months.
I am completely unable to empathize.
At my company offsites aren't mandatory: those who want to go, can go; but those who for some reason don't want to or can't are not looked down upon as "non-team players" or "antisocial". People are all different, and it's about time you recognized that.
If one can't empathize, I'd hope one would at least be able to sympathize with another's objectively different situation.
If you're travelling away who will look after you kids? Pets? Parents? Believe it or not most people have responsibilities.
Ultimately the ideal is not a one size fits all solution but different companies developing different cultures remote/hybrid/in person strategies and employees having options. Seems like the big tech companies are all just kind of copying each other and no one is trying to really differentiate themselves here.