I don't think we "forget", I think developers are after the quick buck and buyers are after most-sqft-per-dollar. Building up or down is more expensive.

Developers just build places for people to live and work. Like the house you live in. They could build a lot more in places like San Francisco and Palo Alto. Yes, it's a bit more expensive, but that's a pretty minimal component of the price at 2/3/4 stories.

The real problem that prevents building up and in is rampant NIMBYism.

Is it? Assume that land is expensive. If I build up, I can create a lot more units for the same land cost (at the probable price of higher building cost per unit). For some land price, there's probably an optimal height to build to that decreases total cost per unit.

Well, it really depends on where. When much of Silicon Valley was built, I believe land was not really that expensive, and that continues to be the case in many parts of the country.