Regarding your anecdote. I definitely notice the same thing, walk through the wealthiest parts of any bigger city and you will see many/most of the houses completely shuttered for >90% of the year. That also explains why the vacancy rates are low, the wealthiest 0.1% (whom ended up with much of the relief funding during the GFC and Covid) park most of their money in assets, but they are not interested in renting out these houses, so when they buy houses they reduce vacancy rates even when they don't live in them.

Growing housing costs and wealth inequality have been extremely correlated, anyone looking for other reasons for the housing inequality really needs to have a very good explanation why they are not related.