I find the opposite. Fairly often one of my comments will get a quick downvote or two, but they quickly then get enough upvotes to get back into the black, I expect as reasonable people, who might not have bothered upvoting the comment otherwise, do so because they can see that it didn't deserve downvotes.