The concern is more how many of those people will show up to vote against you in races which can have very small numbers of voters, and how much of a hassle they'll make for your office. If someone decides that you're part of the problem, you'll be getting lots of contacts, public records requests, etc. The things which circulate on neighborhood listserves, Facebook groups, etc. can be pretty absurd but they might break down into the 95% who don't care and ignore them and the 5% who care enough to put a lot of energy into obstruction.

None of that precludes doing the right thing, of course, but if it's not an issue you care strongly about you might decide not to fight it. The best countermeasure is for everyone else to tune out local politics less, which is an area people oddly ignore even though it has the most day to day impact on your life.