I strongly disagree, and have found myself downvoting more and more lately to (what I've noticed to be are new accounts) derisive comments, one-liners that detract from the conversation (vs. contributing) etc.

If you want to keep your community at a high level, then I feel it's important to be able to illustrate to contributors what you consider to miss that mark. I used to comment along with every downvote, but there are so many bad comments on HN of late that it would take a full time job.