The problem is likely more rooted in wealth inequality. As a smaller group of wealthy individuals control a larger portion of the nation's wealth, they will inevitably acquire all sorts of investments (stocks, real estate, etc) driving prices up.

If there weren't luxury multi-million dollar apartments to buy, they'd be buying the brownstones anyways. And given that the multimillion dollar units sit on the market far longer than the more affordable housing stock, is there even really high demand among the wealthy to own premium housing?

Luxury housing gets built because it's the only thing that's profitable to the builder once you take into account zoning and permits and regulations that need to be satisfied in order to build. What we really need is relaxed policy that allows for the profitable building of run-of-the-mill, average housing.