Sweetheart deals for luxury housing and rent controlled housing mean that developers are going to put more time, energy and money into bringing upper income housing into NYC while the middle and lower income people are left with fewer options.
Because of the geography and municipal structure of the area, middle and upper-middle income housing is doing quite well here.
Just because we can explain why it happens doesn't make those prices any less exorbitant.
Pricing is exorbitant in other facets of life in NYC as well but that is a separate discussion.
It's not like SF is crowded, it's just that we've built really bad buildings for holding a lot of people.
Can you point out a city that built up and is affordable (with a good economy also)?
Here is a random 10-year old condo for $255k: http://www.trulia.com/property/3201711637-222-N-Columbus-Dr-...
Here is an older, further-out one for less than half that: http://www.trulia.com/property/3193730688-3950-N-Lake-Shore-... (still in a decent neighborhood and near a 24-hour rapid transit line)
Here is a large 3-bedroom for $485,000: http://www.trulia.com/property/3197384352-212-W-Washington-S...
Here is a large 3-bedroom, futher out, for $450k in a newly-built brick walkup: http://www.trulia.com/property/3027918821-2550-W-Logan-Blvd-...
There are a lot of variations on similar themes.
Chicago's economy isn't San Francisco's, or New York's, and it doesn't have many exciting software jobs or world-class outdoor recreation. But it is still a big city with big-city opportunities and many well-paying jobs. And it clearly demonstrates that a sufficient housing stock guarantees no building type, even skyscrapers, are inherently tied to high prices. Just as San Francisco's predicament demonstrates that keeping old buildings in place doesn't hold down prices.
From personal experience, at least, supply is _not_ an issue in Tokyo. You can go to a real estate agent and go see like 4 or 5 places in a day and take it almost immediately.
This is compared to (second hand stories) of apartments in Paris where you have 10 people show up to look at 1 apartment and it's a crapshoot whether you can get it.