You are 100% correct but I don't think we can ever get there. There is a huge (not big enough, but still huge) class of normal hard-working people who have >75% of their net worth tied up in a single home. Anything done to punish the upper class of landlords and vacation home owners will naturally hurt the value of homes all over the price spectrum and cause a massive crash in the wealth of the middle class. This will not only affect today's homeowners but the next 1-2 generations of their family. This crash in middle-class wealth would basically wreck the entire consumer economy.

I can't see a way out of this that isn't basically "blow up capitalism and start over". And while I definitely want to blow it all up many days, I also have an extremely strong preference for national economic stability. I really don't want to live through some bigger version of 2008 again.