The problem I perceive is that average score does not say much about a specific comment being non-productive, so someone with a high average score could toss in some complete junk and get it sorted to the top in spite of better comments by average under producers further below.

I'd say that if it remains below the detection threshold it might be a net plus, if too many of such instances appear it might be a loss.

Here is a link to the detailed thread about the experiment of putting user names of high-karma/comment users in orange,

http://news.ycombinator.com/item?id=467181

which as you'll see was a controversial experiment. (I post this link for the newer members of HN who weren't participating when the experiment was tried.)

With their snoots in the air, they would sniff and they’d snort “We’ll have nothing to do with the Plain-Belly sort!”

And, whenever they met some, when they were out walking, They’d hike right on past them without even talking.