"I'd imagine a ton of the people who bought at $500k can't afford $1600/mo on top of their mortgages."

If you have $1M in capital in your $1.5M house, banks will be begging you to take out a reverse mortgage to pay your taxes. It just means that your heirs will get a slightly smaller windfall from your death rather than the full $1M.