Millennium Partners [0] navigated San Francisco's notorious entitlement process. That's what they do as sophisticated developers. Part of that navigation (according to Wikipedia) was refusing to submit the project to peer review. In real-estate development terms, that usually means lawyering up and running rough-shod over city staff and usually garnering project support of line staff's political superiors. Better real-estate lawyers cost more money and city staff attempts to influence every real-estate development whether luxury or market rate (that's their job and why there is an entity called the "planning department"). The important point is there are a lot of fixed costs with real-estate development that only weakly correlate with project size. [1]
More importantly, the money behind Millennium Partners needs big expensive projects:
With these projects, Jeffries created a new investment vehicle to provide equity capital. The resulting financial structure included a consortium of German financial institutions including ERGO, Provinzial Wuerttembergische, AXA Colonia Immobilien AG, Energie Baden Wuerttemberg AG, as well as Goldman Sachs' Whitehall Fund and George Soros and his Quantum Realty Fund Limited. [2]
These are the interests that drive speculative real-estate development and speculative real-estate development is where most housing comes from these days, public housing is mostly done and dusted.
To put a finer point on the lack of deep market interest in building housing, consider Apple's new headquarters. Residential was never even on the table for the parcels Apple acquired from HP. No residential developer was going to outspend Apple for property adjacent to Apple's existing campus[3] and if one had, the housing would have had to have been luxury not market rate to make the numbers work. At the reported $3,000,000/acre, 20 dwellings/acre and 4:1 construction to land ratio each unit would be ~$750,000. Once you're at $750,000, there's no reason to cap unit mix at a 4:1 cost. That's just where they will start.
That brings us to risk. How quickly can Silicon Valley actually absorb 2000 $750,000+ units? What finances $1.5+ billion+ of residential development? Not today. Back in 2010 when housing prices were falling and financial markets were still unsteady. Apple was able to purchase because it had a pile of cash. Residential developers didn't.
The only people Apple was competing with were institutions interested in parking money in real-estate and waiting for the value to go up. In an alternate world where that happened, there is no reason to assume that the HP parcel would have wound up as housing. It might have, but the most likely outcome is it would have remained as the "higher and better" commercial use.
[1]: In residential development there can even be an inverse correlation between entitlement costs and affordability: i.e. an SRO will probably face more political friction than luxury apartments.
[2]: http://millenniumptrs.com/ see Christopher M. Jefferies profile.
[3]: Even commercial developers couldn't make the numbers pencil out for commercial reuse of the existing buildings.
Sure real-estate investors often complain about regulations. But those complaints are rooted in the effects of regulation on liquidity and profits, not on afforadability. Up market housing gets built because that's what people with money want [1] and where the net returns from developable parcels are often higher...commercial development has much higher potential returns still. That's why the big money like insurance reserves often prefers Class A office, retail, and industrial REIT's over single family or condominiums.
The big money in real-estate that drives markets is at a scale where affordable or even ordinary housing deals aren't big enough. The timelines of big real-estate money are often 30-50 years to match liabilities like life-insurance. The only part of the US housing market which comes close to that timeline is conventional 30 year mortgages. But at $100-200k per deal, it takes thousands of deals just to get $1 billion out the door. Then the thirty years is still just theory: there's refinancing and home sales making the practical investment turnover five or ten years (times thousands of investments).
Unaffordablility is a sign of high market demand where affordable housing is less preferred by investors than raw land or unredeveloped redevelopable property. Holding developable property means there's a real asset beneath the investment and the risk/reward can be spread across multiple economic cycles. Construction not only requires the injection of additional capital, but ties the risk/reward to a particular economic cycle. Again, it is worth keeping in mind that at the macro level the real-estate markets are largely driven by parked capital not borrowed money in the form of money. B2C isn't what drives real-estate markets over the long term.
The US housing market is driven by government intervention. FHA mortgages and the mortgage interest deduction are two easy examples. These exist because traditionally there has not been much incentive for the bulk of capital in the real-estate market to invest in houses for ordinary Americans. Historically, the sprawling checkerboard suburbs are largely the result of smaller ad hoc deals not the really big players. Though that's changing a bit, the involvement of the big players tends to encourage housing to move upscale and into multi-use developments with commercial and retail components..."the affordable lifestyle" doesn't look good on a sales brochure.
[1] Selling to people with money is a good sales strategy.