It's because these housing shortages are paid for by everyone who doesn't live in the jurisdiction and cannot vote for them, so there's a perverse incentive for incumbents to treat shortages as free money that comes from nowhere.

It's not unlike upper atmosphere pollution, as if every local community were able to have their own policy on it. The cost of pollution is paid for by no one in particular but principally other people far away. Each community's individual impact is tiny and easy to deny, but they all tend to choose the same thing which has a greater collective impact. Aren't their rights to choose more important than any cost to society, without question?

Clean air, clean rivers/oceans, controlled CO2/GHG, banning leaded gasoline for highway use, and similar items I think of as different in a critical way than zoning or local housing shortages. There's no amount of money (or other resource) that an individual can expend to buy clean oceans, air, or lower GHG. There are no practical alternatives.

There is an amount of money that an individual can spend to buy a house in Palo Alto. That amount of money might be higher than some people would prefer or elect, or (in many cases) even be able to spend, but there are practical alternatives in any case (to include living in other communities).

In a case where no practical alternatives exist, I agree there's a much greater cause for collective input and decision-making (to include at a global scale in many cases, as these aren't even a state or nation-state level of issue, though it's the states and nation-states that enact policies and laws to implement/enforce collectively-agreed treaties on some of these topics).

Where practical alternatives exist, I find the most local government level to be the most desirable one to make the decisions that disproportionately affect the people who live there over people who don't live there but might like to.