There's definitely a downvote squad. I've noticed my comments tend to do much better their first few hours and downvotes tend to come later after the submission has fallen off the front page.