Yes, it's fine. You bought your lot, not your neighbor's. Zoning and land use laws are one of the primary reason why America has such shitty yet simultaneously expensive cities.