Even if we accept the premise that the housing market mysteriously transcends the rules of supply and demand, the pidgeonhole principle is going to come into effect. How is San Francisco going to accommodate 5-10 thousand new residents each year when only 2-3 thousand units of housing are added each year. Even if we ignore price, there just isn't enough space unless we build more housing.

I'm saying that the rules of supply and demand are not as simple and as clear cut in reality as they are presented in Econ 101.

The simplistic view is correct almost all time. Given a fixed demand curve, more supply leads to lower prices and less supply leads to higher prices. Exceptions can exist in special circumstances, but they are rare and usually short-term.

Not when it comes to human behavior it's not. The simplistic view tends to assume everyone is a perfectly rational actor, and that there aren't other forces at play.