There is a fixed amount of land available in a city, and a city is responsible for servicing it. If you've a property that's gobbling up some of the city's carefully-managed land and creating an eyesore that's a detriment to your neighbors, then it's perfectly reasonable for the city to take action.

You are a part of civilization. Accept it.