I agree with you on this principle, and when your property does not affect anyone else this right is an absolute. However groups of people who choose to live and work near each other generally decide that they have some say in what can be done, and this usually works out for the greater good.