If we make the (extremely crude) assumption that the capital expended on that 2nd home could be moved elsewhere, we could have built more housing in cities instead of building and maintaining that home.
[0]https://www.google.com/amp/s/www.statista.com/chart/amp/1963...
Supposing everyone had more money in their hands (all else remaining equal), that would just be reflected in the market. Houses would increase in cost. Now you're where you started.
Housing supply is just too inelastic in most US cities.
At any rate, the point is that qua "second homes", they represent a drop in the bucket compared to the overall market. Home ownership rate is around 63%, for a population of 334 million. The inelasticity of housing construction whilst demand is steadily climbing due to immigration has far more impact on average housing prices than cases of the ultra-wealthy constructing or purchasing a mansion upstate.
Those mansions also tend to be built far away from city cores, with large lots to maintain (with paid labor), more taxes to be spent for infrastructure cost, requiring a vehicle, etc. It would still be too expensive and unhelpful for average people.
WE don't build enough. Look at what zoning reform has done for Minneapolis.
More housing starts means more affordable homes. We've seen what zoning reform has done for cities. We don't build enough.
Investors collectively do purchase a good amount of new housing starts (like 18%?) but that's also spurred by interest rates and confidence that demand will grow. It's neither here nor there, build more and houses become more affordable.