New buildings require a much longer minimum investment timeline because the building doesn't exist yet. It's also far easier.
So if you let the free-market write the rules here, you'd get essentially exactly what we already have, because that's essentially exactly what we've already done.
Instead, if you really want a free-enterprise approach that encourages better land utilization, you'd need to look before you leap. For instance, a conservative rent and property resell value control would guarantee a maximum ROI for the buy/hold and do nothing/dump strategy. Otherwise, you'd have to indirectly subsidize new developments through tax breaks and other financial incentives (which is exactly what is happening)
Alternatively, if you're willing to forgo the engineering and manufacturing of markets you could use community land trusts, which is likely a far simpler and less issue-prone model of execution. I've personally invested in a few and have seen a long-term, stable, dependable 3% ROI (which is better than other equally conservative vehicles) so this doesn't mean that capitalists and investors are gone - just all the messy parts.
Supply and demand :)
The state can't tell cities to 'build' - as cities don't do the 'building'. Private contractors do.
Methinks they could make some state-level zoning laws or something ...
But if there's incentive to build people will build.
If there's not a not of building in the Valley ... it might be due to zoning and regulatory issues, perhaps they could streamline? :)