I do the same; a while back there was a discussion on HN (approximately): "what do you do when you see someone being downmodded unfairly?"

Up/down votes on comments should be, in my view, for whether you thought the comment was worth reading or not, irrespective of viewpoint. Likewise for upvotes on stories.