I think one issue is that there is no way to express disagreement with the worthiness of the original story. No downvote on stories, no bury etc. So instead some people most likely express that by downmodding any comments that agree with the storys worthiness.

I'd really like a downvote on articles, although it's a tricky decision I guess.