Well they don't have any rights over property they don't own. Why should a narrow majority or as is often the case a small but highly motivated minority have a veto over any conceivable use of your property just because it happens to be near them.

I'm not proposing allowing a chicken processing plant in a residential area, but it's no one else's business if you want to build a duplex on a lot you own.