It’s important to focus on solutions that solve both sides of the problem, and to deny arguments that it’s functionally not possible (which would be an unacceptable outcome).
I don’t mind the contradictory positions in terms of market theory and efficient allocation, and believe it’s important to pursue anyway, that this technical contradiction is fine... necessary even.
an outcome can be unacceptable, but you have no choice but to accept it. It's like gravity will pull down, whether you accept it or not.
When you lose the ability to evict, you also lose the ability to rebuild (since you'd need to evict to rebuild).
What is desirable is balance. Where that balance lies is up to the populous, and in san fran, the populous have spoken (by their actions) - they prefer to keep it less dense, and suffer the consequences of high property prices.
These two ideals of yours are contradictory. The right to not be evicted (in order to better develop the land) can directly prevent the availability of housing with the qualities you require. With the help of zoning boards, which are really just an embodiment of their constituent land owners, isn't this more or less the sum of the housing issue in the bay area? You certainly cannot say that there's any semblance of comfortable hosting with a reasonable commute...