And yes they would keep building. Land owners are the ones who gain/lose by fluctuations in demand, not developers. If demand drops and rental prices drop, land owners lower their prices that they sell to developers. This has been proven so substantially by Housing Economists that it's basically considered consensus by now. Spend some time with the Journal of Housing Economics if you would like to learn more.
The real reason why developers only build luxury buildings is that they can't build enough to satisfy all demand, so the few buildings that they can build will obviously go to the types with the highest margins. If we allowed for more development, developers would still build luxury buildings as long as there was unsatiated demand for them, but they'd also build normal and affordable housing as well.