Vacancy is largely a function of the relationship between supply and demand curves.

What it also shows is how open to changes a given market is; if there's too much vacancy (think Detroit) then it's clear that something went VERY wrong in the market.

If there's too little vacancy (think SF, Seattle, etc) then what went wrong is mostly NIMBY and regulations failing to pair jobs and housing increases together. The result is a lack of flexibility and skyrocketing prices.

Yes, that’s exactly the point. Only supply and demand matters, density is no more relevant than any of the other factors that influence supply or influence demand. And within the abstract algebraic model of supply and demand, and density only being something that increases demand, increasing density never increases prices. That doesn’t mean that dense areas can’t be expensive; that much is obvious. It merely means that there are other reasons related to supply and demand that made them expensive.

Why are you so certain that none of those component factors could relate to each other in more interesting ways? Why say "it's just supply and demand" when it would be more interesting to look at what drives supply and what drives demand?

Jobs create demand. Jobs bring skilled workers. The presence of skilled workers itself attracts more employers. Maybe the employers move in faster than upzoning happens. Maybe it happens slower. Whether or not this increases or decreases price is just a question of the coefficients on the various inputs. I'd still propose that the missing piece is geographic - cities that can absorb this by growing outward in a decentralized way will stay cheaper than ones that can't, where things get nasty.

But, to move the discussion along, let's just call it demand outrunning growth in supply, dropping the precise role of density (or replacing density in my original post with "geographic constraints which necessitate increased density" or something). Either way the result is the same thing. Do we have ideas that we could try to get turned into policy for stopping this sort of thing from happening? It's a really common pattern, not just in SF but everywhere that's ever seen someone complain about gentrification, so it would be a useful thing to have a politically feasible solution for.

EDIT: it's getting late out here, so I'll add one last bit then probably retire for now. I personally am not a fan of remote work, or cross-timezone collaboration. But I think the people pushing it need to win out in the long run. When it comes to massive density increases, or de-densifying the country by spreading work back out thanks to technology allowing better and better telepresence, I think the latter is far healthier in the end. It de-SPOF-ifies industries, which is good when considering natural disasters, potential climate impacts, or war/terrorism. It levels out costs - you now have the whole country (world?) to choose from - while also massively increasing your hiring pool. It evens out opportunity - you don't have to be able to afford to take a gamble on moving across the country to chase your dream job. It saves a shit-ton when it comes to energy used for commuting.

And if you ask me if a less dense, remote-friendly world is more likely than getting people to reverse course on a hundred-year-old car-powered trend towards suburbanization, I'd wager yes. Especially because the arguments for the latter are pretty unconvincing so far for people who don't want to live somewhere massively denser and who are able to afford their current living situation.