You are referring to properties held by the already rich (only the rich can afford 'vacation homes').

If they can't afford a 'vacation home' without short term rentals then it should be sold and provide housing year-round and build equity for a new homeowner.