If you think about it, zoning and land use laws are just the codifying of things that are detriments to your neighbors. Obviously, no one is going to agree with all of them, but they are what society has deemed worthy of forbidding.
Only if by 'society' you mean rich old rentseeking stay-at-home busybodies with nothing better to do than complain at town council meetings.
Edit: less snarkily put, the people doing the forbidding are not representative of society and are not acting in our collective best interests
Pretend there is a dark shroud covering away all boundaries of your property line. Whatever elements, whether noise, shadow, smell, smoke, etc enter into your property volume, you are allowed to complain about. Anything outside of that, you are not. How damned hard is that?
Do you have any examples of zoning or land use laws that don't fit your definition of 'things we are allowed to regulate'?
We needed to submit drawings of the garage, explain that it fit well with the main building, a distinct house from 1909, well known in the neighbourhood. Describe the colour to be used, including the colour on the garage door.
They come and check that the foundations get placed within four inches (10 cm) of what the building permit allows.
Do I mind? No. End result is good, took longer than expected due to heavy load on the city planning department.
In some villages in Switzerland you get fined if you don't have the right colour flowers on your balcony. (Or maybe the flowers showed up on your doorstep, with a mandatory bill, without having to ask.) :)
Edit:splng
Given that the two condos that sold near me (one next door, and one two doors away) went in the 300k-350k range, it doesn't seem to be depressing the property values much either.
If there is a specific noise/shadow or other detriment that is clearly caused to an adjoining property, then OK, the adjoining neighbor should be able to stop it (or sell the right to complain as an easement if they wish). If there isn't, and it just "looks ugly" and "isn't normal" then there is no meaningful detriment and they should be allowed to build