I've seen sometimes that my downvote doesn't count (doesn't cause a fairly new comment to go gray, and to be even more certain, doesn't lower the poster's karma).

I've experimented quite a bit with it and have some theories about when it works and when it doesn't, but it's still not completely predictable to me.