If coming up with an objective measure of quality was feasible, the voting system could be scrapped entirely, and instead of making a site that allows users to submit links, pg could have written a web crawler.

But web crawler doesn't have an important input: how HN users would vote on a particular submission.

Sorry, I don't get your comment.

I'm just pointing out that it should come as no surprise that the author of this blog post didn't come up with a good metric for judging quality. As in, if the author had come up with such a metric, he would have started a company and become fantastically wealthy applying this magical quality-quantifying algorithm, rather than writing a blog post.

Perhaps. But they are degrees of usefulness, and it's probably possible to do better. Especially if you do an analysis in hindsight, where it's no longer possible for the participants to game the system.

Yeah, that measure was super hacky and I have no justification for it (but it was the simplest/only thing I could think of at the time). Do you have any suggestions?

One thing I'm thinking of now is seeing how (# points at the xth percentile) / (# points of 10th highest rated post) changes over time. (Or maybe I should just take a closer look at how the overall distribution of points changes to get a better feel first.)

There is no way to get a handle on comment and submission quality without content analysis.

You need a list of properties that define good comments and good submissions, you need some way to quantify those properties and then you need to invest obscene amounts of work coding those properties.

All of this is hard work and full of uncertainty. Different people want HN to be different things, there can’t be one list of properties that does them all justice. Who gets to define what a good comment or a good submission is? Is it even possible to exhaustively translate someone’s understanding of a good comment or a good submission into a list of properties? Can those properties be quantified, in the best case with as little work as possible? Does the coding process scale to the amount of content that has to be analyzed?

This is a Ph.D. thesis worth of work. I wouldn’t try to do it as a hobby.

> This is a Ph.D. thesis worth of work. I wouldn’t try to do it as a hobby.

Why not? They won't hurt themselves, and it will be fun. Maybe they will actually learn something.

I wouldn't. I don't object to others trying their luck ;-)

I don't know how to come up with a statistic that reflects quality of comments at the top of my head.

But the statistic you chose is an interesting one. As is the distribution of points.