That may be true if the limiting factor is the number of developers - they will only develop the highest-value properties. But in a city like New York the limiting factor is not developer capacity, at least not that I have ever heard. Developers are effectively constrained by the number of profitable building sites, not their capacity to put up new buildings. And, in turn, the number of profitable building sites is dependent on not just the cost of labor and materials, but the cost of navigating city planning, obtaining permits, etc.
Don't forget that land is cheaper to acquire in less desirable neighborhoods, and it is also generally cheaper to build there due to lower existing density (which eases logistics by making it easier to get equipment in and out, allows more on-site storage, etc.). The development that maximizes returns is not always the development that maximizes revenue - cost and time also matter. And there are more developers who can handle building a 10-story building in the Bronx than can tackle an 80-story mega-project in Manhattan. Plus, a "cheap" neighborhood in NYC is often as expensive as an "expensive" neighborhood in most US cities, so if developers are willing to build in Milwaukee or Cleveland, they should be willing to build in the Bronx.
But in a world where the administrative cost of starting a relatively small project is overwhelming vs. the potential returns, then those smaller projects in lower-price neighborhoods simply won't get built.
Net result is that a developer can profit quite a bit more from luxury housing than affordable housing. If you are a developer (which presumably means a for-profit business) and have a choice between the two, you naturally decide to go after the higher profit option. Can't say I'd blame them either.
This is where incentives need to be provided to balance the profit equation. If they could make the same amount of profit, or gain other business benefits, I imagine the problem would start solving itself.