It's a mutual agreement to strip each other of troublesome property rights. These people aren't exempting themselves. They want to prevent tragedy-of-the-commons troubles, spite houses, and other anti-social troubles. They mutually agree (government) to prevent anybody from making the place miserable.

Thus there is no right to build: an 80-story apartment tower, a supervised injection place for IV drug users, an organometalic peroxide production plant, a hog farm, and a tire recycling plant.

The demand is there for even more restriction. In many areas, new housing is exclusively in home owner associations or in condominiums.