I think there's a tradeoff. Smaller cities should be more affordable, but there is less labor mobility, leading to higher economic risk. If a person finds themselves in even a moderately specialized occupation, and loses their job, there might not be another job for them unless they move to another city.

Thus they either have to deal with periodic un- or under-employment, or manage the risk by building a cash cushion, either of which means a less affluent lifestyle.

But what happens when very few of those professionals can afford to live in the big ones?

The price goes up for folks living there? And then plumbers can afford to live there. Its good old supply-and-demand freemarket principles at work.