But they compete as governments, rather than as direct builders. That makes it really hard for it to work as a market.
Cities can and often do act as independant (global) agents (bidding for Olympic games comes to mind, and offering special sweetheart deals to banks etc). London has a revenue and expenditure of roughly 22BN - putting it around 150th in the Fortune 500, and it is far mnore concerned with attracting and retaining talent from New York or Paris than from Sheffield.
If London fails to provide the right mix of house building and amenities, those people and businesses will (eventually) go elsewhere.
Humans do not physically move often or easily - but cities do rise and fall and compete with each other - they are the right level IMO to look at "good neighbourhoods" and if we want to solve the "livability" of cities, then cities (or possibly sub-city-areas) are the right sized unit for the discussion.
Its not about houses, its about neighbourhoods.
There is a problem of indirection, as chez17 alludes to. A "city" is too much: a place to do business, a place to live alone, a place to explore, a place to retire, and so on and so on. Some of these measures are so critical that failure can be crippling. Thus, a city might compete for the attention of international organizations like banks and the Olympic committee... but why? What's the actual value exchange happening there and how does it play out in the details? (I'm not asking "why should they", but "why are they".)
In answer to why - tax revenue is one less than compelling answer - my preference is to look back at the city states of Greece - they competed and fought for supremacy, for survival for riches - but mostly they fought because their city was their tribe and their home and the others were enemy. It's pretty human basic stuff I believe - nations are too big to get really worked up about but a city - it's the right sort of size.
It's not so much the city as the jobs a person can get. If you get offered a job in Boston or London, the odds are you're not moving to New York no matter how much you like it. Cities do compete a little bit (you get an offer in Boston and London), but overall it's almost exclusively where you can get a job.
And areas of cities (I think the term is metro-area, including feeder towns etc) compete within the city - Kent is a lot more expensive than east essex and both are roughly the same distance from London.
I am not saying the micro level analysis is wrong - just that only considering commercial house builders as the only agents with skin in this game is too narrow.
Each city mayor will adjust their regulation mix to get the best out of their towns - if we don't like the results that's probably because local democracy needs more effort from us than traditionally it gets - every city could be a pedestrianised beauty if getting elected depended on it.