I tend to downvote very rarely and only for clear violations for the rules, not for comments I don't like. Telling the author why you didn't like something they did often gets them to change or explain their behavior. Just because something is upvoted doesn't mean it is something that should be on Hacker News.