Unfortunately a lot of folks think that housing doesn't follow the same simple economic rules that we've observed for all other physical goods. So they don't allow supply and we continue to spiral deeper and deeper into a housing crisis.
If housing were abundant, we wouldn't be reading articles about the rise of living in cars.
Further, there isn't unlimited demand at every price point. When you constrain supply to the point that high-end demand can consume more than total possibly supply, of course builders only build to the high end of the market.
Finally, we've made the high end of the market the only thing which can possibly pencil out by driving up costs. Planners demand low floor area ratios, high parking rates, articulated facades, parks dedications, high impact fees - that money ultimately comes from the buyer or the renter. So by making those demands you price out increasingly large portions of the population.