EDIT: While I’m inside the edit window, let me throw in another big variable as an example. Costs are another factor that inform a price, and a big one. You could make a case that it deserves higher billing than supply and demand, but if your costs are $X but you can only sell your widget for $X-1, and not $X+>=0, then $X-1 is your price ceiling. Your business is unprofitable and losing money.
I’m trying to be careful with my words but I still seem to be leaving some people with the impression that I think supply and demand is the only variable that informs a price. It’s not, but it’s still incredibly important and often dismissed as a non-factor when discussing San Francisco rental and real estate prices with San Franciscans.
If landlords can make a killing by constraining supply politically, they will. So the primary cause is the incentive system and the political environment which makes it possible for them to do that.
It's all about how the reward functions are designed. So of course if you reward scarcity and profiteering that's what you get - as a secondary outcome, not a primary cause.
In cities with zoning constraints, developers are the only natural predators of landlords. They are entirely different skillsets, building new units requires managing an extremely difficult political process for permits, raising capital, working with contractors and labor, etc. Being a landlord requires very different analysis and skills.
There may be a few landlords with the capacity to cross over to development too, but it isn't many of them.
If you have a bunch of left wing groups who are anti-development fighting to keep supply constrained even while complaining the rent is too damn high and the character of the City is changing and yadda yadda yadda, who are you[1], as a landlord, to take the wind out of their sails when you are profiting from their protests?
[1] The general “you”, not you specifically.
I've literally never seen it dismissed as a non factor - not even once.
Supply and demand is pretty obvious because of the basic physics behind it. If you have a world with 10 houses and 20 people then 10 people will be without a house. It's not very difficult to see the problem and how to respond to it. We first have to establish that we want every human in our country to have shelter. That means what we don't want to happen is the allocation of housing to a subset of the population, be it via the free market, a lottery or an application process where you are vetted by an interviewer according to some criteria. We don't want any of that. So the obvious solution is to just build more housing, regardless of the economics.
Now lets just add a thin layer of economics.
It turns out that the free market has this interesting property that in the case of demand exceeding the supply the cost of housing increases until it becomes profitable to build more housing. It's a self balancing system if you make sure that it's not getting stuck somewhere.
Turns out if you make it really difficult to build more housing then people decide to pack up and leave because that's the only option left. For the sake of the argument 20 people can't live in 10 houses so 10 people will have to go build housing somewhere else.
Now here is the perverse thing. The same people who made it difficult to build more housing also made it so that they get to stay where they are by removing the pressure to leave the location. What we get is 10 people living in housing and 10 people living in misery. You don't need an economic resource allocation mechanism like the free market to cause misery. The misery already exists in the real world and the economics merely summarizes that misery as a price signal.
So why care about supply and demand? Because it's the closest thing to physical reality but what people fail to grasp is that you have to change physical reality to change the economics. No, messing around with prices via rent control or prop 13 does not solve the underlying problem.