The places where people don't go tend to have more of them. People don't go there for a reason. The entire South and Midwest have places where people do go.

What you don't have is a paradise where everything is cheap, there is a lot of employment and other opportunities and people somehow refuse to go to in the long term. You have cheap places where there is basically single major employer or low earning industry.

I am not saying that people always have it perfectly right, but in the long term and on average, the mass moves where it makes sense to move to. Those places then become more expensive.