I think there's a tradeoff. Smaller cities should be more affordable, but there is less labor mobility, leading to higher economic risk. If a person finds themselves in even a moderately specialized occupation, and loses their job, there might not be another job for them unless they move to another city.

Thus they either have to deal with periodic un- or under-employment, or manage the risk by building a cash cushion, either of which means a less affluent lifestyle.