Just because middle class people benefit from the subsidy of low interest rates that create asset bubbles does not mean that the people who raised families in a neighbourhood or a city are somehow undeserving of the rewards on their equity.
The idea that money not taken in taxes is an "expenditure," or an advantage, and this idea of accounting for the hypothetical opportunity cost of not taking some peoples money as a tax privilege is bizzare.
Yikes. Do you really think places like SOMA, SLU, DTLA, etc. got better because rich people brought their 'culture' there? I'd recommend you visit said places and see for yourself, most of the 'culture' is in adjacent (usually historically minority) neighborhoods.
These are the people that brought their culture. Their housing is probably also well appreciated.
In a lot of markets this absolutely hits the "moderately wealthy trying to leave the working class". Bay Area houses that went for $1.2M in 2009 now go for about $3M, for a gain of $1.8M. That's well over the $500K exclusion, even including capital improvements.
Few folks will shed a tear for people who own a $3M house simply by virtue of living in a hot area, but that's exactly who the OP is talking about.