If workers couldn't afford to live close to those cities they'd move somewhere else. Those cities would then experience a shortage of workers. Wages would have to rise to attract new workers. Costs to everyone else in those cities would have to rise in order to accommodate this. If non-workers could no longer afford these costs they would have to sell their homes and move out. This would increase supply and bring the value of housing down in these cities.

Workers can’t afford to live close, and they haven’t moved elsewhere. Why? Perhaps moving costs money while they never have an opportunity to save for it.

Unfortunately the places where workers congregate is also the places where the most opportunity can be found in the job market and social sphere. Pushing people out of our most dynamic cities so that a bunch of landlords can artificially restrict supply to profit more is idiotic and a failure of public policy.

Assume a frictionless turtle.

You end up with a permanent fight for the impoverished or less well off to move to the city to maybe get some table scraps.