This is an "easy" fix in theory: gut the previous system and just build anew. The problem is, that would be such an enormous undertaking (b/c you have to divert sewage during the whole process, etc) that the city is effectively forced to apply what amounts to band-aid fixes in its place.
It's definitely not win-win if that shadow falls on your beloved back yard, ruining your legal cannabis plot. Or if you live in the top floor of the five-story building that was built prior and your view will now be of the six-story building.
NIMBY works ("works") because most people put their own specific self interest before community when the former is threatened. They do this pretty reliably, and on the individual level it's usually pretty rational.
(Also, I might have reasons other than economics to keep my 1500 sq ft house from being torn down.)
The answer, as mentioned, is go to a higher level of government where there are even more people to outvote Mr. “But Muh Views”.
In the case of CA there’s seven million people in the Bay Area and 33 million in the rest of the state that would benefit from continued growth there and don’t care at all about shaded backyards in Palo Alto. That sounds like a referendum waiting to happen to me.
That would come up against the same forces that fuel NIMBY-ism, i.e. the relative political clout of home owners and the power of the story that it's my back yard today, but it's gonna be YOUR back yard in the next shadow.
Bingo. If there is currently a detached house of 1500 sq ft that can be torn down and in its place can be put up a six story building apartments totaling 8000 sq ft of livable space, there’s plenty of room for profit for the prior homeowner, the developer, and also for the new units to be more affordable than what was there before. Increasing the supply is a win-win-win.
As I said, if necessary—-and it looks like it is—-higher levels of governments should step in to prevent local, narrow minded, parochial interests from killing the goose that lays golden eggs.