Is it a mechanism to find the best comments in a thread? Is it a visualization issue? Is it ego? Is it a voting mechanism to know what others think is important? I honestly do not know.
Yes, it's a way to find best comments in a thread (not guaranteed, but much better than nothing).
Has nothing to do with ego.
The nature of systematic conversation ensures there will always be groupthink. Systems are designed to function in certain ways and deny other approaches to get their desired result. The result is that people who want this system flock to it, and people who want to somehow act counter to this system are going to be a minority that's possibly chased away by the system itself. Options like downvoting and "ranking" comments" and flagging inappropriate topics basically say that there is a right way to behave or talk, there are topics that are better than others. Then either people are attracted to the sites that reinforce their worldview or they're conditioned by these rules to decide there're right and wrong ways to think or to be – I'm not sure which it is. Probably a little of both.
But if you had a button for each piece of information it would look like
[] well written
[] well thought out
[] agree
[] disagree
[] poorly written
[] funny
[] off topic
For bonus points they would be in a random order in the UI so people wouldn't develop muscle memory voting on comments.As for more detailed buttons - I think that would make it too complex. Slashdot has established the same concept, probably based on the same reasoning (mostly to distinguish between "just" funny and actually insightful comments). Can't speak that much about how successful it is, maybe somebody else has more insight here.
I am working on a site now that is heavily based on passive activity voting, users have no idea how their actions are being used by the site to rank itself. Their inputs have much less weight than purposeful actions where both parties are aware of the event.
It is almost like you need an agent that votes in certain public ways (sentiment analysis,summarization,etc) to decide if comments snarky, funny, off topic, etc.
Passive voting sounds interesting, though - would that work via Javascript? How would you determine what the user is focused on?
Finally - I think it's important not to get ahead of ourselves - because I think that's what happened at Slashdot. Talking about new classifications is all nice and well, but it's useless when you don't have a clear plan what it should result in and how it would engage people. You need a clear use case.
For instance - my proposal about an off-topic discussion button is pretty straight forward. People would see the immediate benefit: Hey, this can help us make conversation less wasteful and save me scanning time as it separates the cream from the crop. I think it wouldn't have a problem enticing engagement, too - nothing drives engagement in nerds like being annoyed, so they'd click that button.
On Slashdot, I can theoretically reduce a conversation to exclude all the funny-only comments so I'm left with just the cream. But seriously - how often does anybody use that? Once you give people elaborate personal filters, you actually end up having them worry whether they're missing something. Which is why they can only be consistent and transparent. If I reduce a comment thread on Slashdot to exclude the funny business and see that one of the nodes has a HUGE conversation, I wonder whether I should check it out and maybe end up annoyed because it's just a chain of memes after all. If there was a way in HN to mark things as off-topic and have that a clear process that is always applied, I think I would be more confident in skipping. Mainly because if it was broken, the community would complain until it is fixed.
I don't have a solution. I am not sure there is one or that it could be "fixed"
Maybe semi-intelligent Eliza like agents that constantly clean the place up.
Stackoverflow has done a pretty decent job I think. Wikipedia could use some work.
----
I think the base set of rules need to be codified by the system. The users can guide each other and the system, but the system itself should be stable.
That's what I like about HN. Groupthink is indeed present (especially when it comes to technical topics such as programming languages or OSes - we all have a strong incentive to promote what we use), but not nearly as much as any other online community I'm aware of. There's a recurring pattern that I like here: usually, when a submission argues X, the top comment in the comment thread criticizes/refutes X. By hearing the two sides of an argument, one can better form an opinion.