Do we care much about the price of land or do we care how much it costs for someone to live there? I believe it's the latter.

So yes the price of land would go up, but the price of livable space would fall. And the cost of livable space is what we care about with regards to affordable housing.