High cost of living areas push people away when the local job market slumps. So, the equilibrium does end up relating to high housing costs just indirectly.

Aka, if they paid less the local population drops which would reduce supply.