There are places where you can't afford to live even if housing costs zero - if "noone wants to live there" then after a single generation, the place has no jobs, no nearby infrastructure (schools, social services, etc), and all the goods and services cost more (if available at all) than in the affluent regions simply due to lack of economies of scale.

This is a big problem in some areas, where urban regions have a lack of housing despite a construction boom, while rural regions have empty houses available for 20-30% of what it would cost to build a similar house.

If I moved to a place like that, the extra cost in time and fuel in driving myself and (especially) the kids between there and a reasonable city would be much greater than what I currently spend on housing.