I reserve downvotes for comments that harm the discussion. Usually this is because of spam or needlessly inflammatory remarks. Though some people have managed to present arguments that are not otherwise objectionably but so utterly wrong I'll downvote them anyway.