Is the argument that people shouldn't be able to agree to a contract saying, "I will not subdivide this land, and will only sell it to someone who agrees to the same condition"? I'm not entirely averse to that, but it seems like it's a more complex issue than just allowing whatever the market demands, because there's market demand for these restrictive covenants (people really do sometimes want reciprocal agreements with their neighbors about what each will do with their land), which would have to be prohibited by restricting what kinds of contracts people are allowed to sign.
There is a proposition in the law of property that restraints on the alienation of land are to be avoided whenever possible. restrictions on alienation reduce the fungibility of property and increase transaction costs, reducing efficiency. In some states, e.g. NY, there are limits on what sorts of restrictions on alienation courts will enforce when written into deeds.
You actually get that in the private sector as well, if a municipality doesn't set such rules. The neighborhood I grew up in in Houston had minimum lot sizes and restrictions on subdividing houses for rental, all implemented via contract law. When the subdivision was built, all buyers agreed to set up a homeowners' association and certain rules, which are now conditions attached to the deeds. In an alternate world where Palo Alto had been set up that way N years ago, with subdivision organizations and contract law mandating lot sizes, rather than municipal government setting them, the end effect seems like it would be largely the same, so it doesn't really seem like a government vs. market distinction (at least, unless you restrict freedom of contract enough to make this kind of private-sector zoning impossible).