The exception, at least for me, is top level comments. They have so much power to shape the conversation that even a merely mediocre one should be downvoted.
Sure, it's possible to comment with a correction (people did), but it's an absolute waste of a comment thread to have the first experience be a bunch of people debunking gibberish. Downvoting plays an important role in sanitizing stuff that's going to waste time and prevent good discussion.
7. You may think you are able to vote, but it may still be ignored
To expand on OP, from what I have observed it's not just related to the newness of a comment, because I have seen new comments with downvote button right next to a slightly older comment with no downvote button (just for clarity, none of the five conditions above apply either, e.g. it's not a reply to mine, it's not already dead..). It also seems to affect top level comments more than others. The mystery remains.
Edit: This thread has been banned from the front page. It was on the first page two minutes ago and now it's not even on page 2 or 3. Can only be found in Ask section. Why?
As far as I know it's almost always user flags.
This is a meta thread, and so people flag it.
#4 interesting. That is thinking a few steps ahead.
1. You have too low a karma (points, look top right)
2. The comment is a reply to yours
3. The comment is quite old
4. Something vague about a mechanism that stops you stalking someone and downvoting (spread the downvotes, don't focus them like a meanie)
5. It's already been downvoted to nothing (you can do it, but I think it has no effect)
Generally, upvote the positive rather than downvote the negative.