Owning property doesn't let you alone change its zoning. If I was your free market property right loving neighbor I would be upset that you're running a hotel in a residential area.
The fact of the matter is that you've been living in a multiple-family dwelling with three residences, only one of which was being rented out. You didn't like the arrangement, and decided to revert it to a single family home, leaving the town with two fewer housing units.
In the process of rennovation, you have been renting it out in the manner of a hotel by layman's standards - but you aren't technically a hotel. Basically, you get to take advantage of a poorly written law.
Property rights only go so far - once you start affecting others, including renting a portion of house for any length of time - you start picking up responsibilities. Free markets? Free markets work well in some areas, some not as well: Others need regulations to various degrees.
And hotels and housing are one of those that somewhat need regulation because we know some folks do bad or stupid stuff to the detriment of others, and what you do as a landlord renting in any capacity affects others.
And you are still renting it out in some fashion. Why should the laws to either hotels/motels/b&b's or laws applying to a landlord not apply? For the most part, such laws are designed to protect a renter and keep minimum standards for cleanliness and other such things. Some rental laws are there because of shortages and prices [1]. Hotels often pay extra taxes. In some areas, a bed and breakfast has looser and different standards, partially because it isn't unheard of to take a family home and turn it into a B&B, with the owners living in a portion.
And truth is that while I'd support someone renting out their place for a week or two once or twice a year without many issues (permit or fees, mostly), once you are regularly doing such, you are either a landlord or a hotel manager and should have such laws governing you. The pushback from those doing this I see as basically folks trying to get out of complying with the law.
[1] Understandably, two units hardly affects the housing market so long as it is an isolated incident, but if or when that trends to multiple folks, it can.
I'm changing it back. I own it. If for some reason the town tried to stop me from doing short term rentals, I would just convert the house to single family and sell it for twice what I bought it for.
Unless one is opposed to property rights and free markets, I don't see how how one has standing to complain.