News for hackers. Stuff that matters.
Re:How Not To Sort By Average Rating by NathanRice on Tuesday April 03, 2012 @11:10AM (#3793223)
I don't think you have to resort to any overly complex machinery to achieve similar behavior. The simplest approach is to just use a non uniform prior. His pessimistic bound could be emulated by having an initial alpha that places more weight on low star ratings. The intuitive interpretation of that being "things are probably bad unless proven good" roughly. Another option would be to generate the prior based on the posterior distributions of other items. Just take the distribution of ratings observations for all products of a given type (perhaps only items produced by that company?) to get a sensible prior on a new item in that category.

The strength of priors here is that it is very easy to take intuitions and encode them statistically, in an understandable way. Taking the lower bound of a test statistic doesn't admit much in the way of intuition.

Re:How Not To Sort By Average Rating by _delirium on Tuesday April 03, 2012 @11:19AM (#3793257)
I agree the lower bound of a test statistic is a pretty indirect way of encoding intuitions. Somehow I tend to find loss functions the conceptually clearest way of encoding preferences about inference outcomes, though. But, in this case my feeling in that direction isn't very strong, and the priors-based solution seems fine.
Re:How Not To Sort By Average Rating by moultano on Tuesday April 03, 2012 @01:05PM (#3793784)
There's a distinct difference in the asymptotic behavior though between the lower bound and the prior. The lower bound goes to the mean as 1/sqrt(n), the prior goes to the mean as 1/n.

That makes for a pretty significant difference in practice, and I'm not sure which is preferable.

Re:How Not To Sort By Average Rating by NathanRice on Tuesday April 03, 2012 @01:30PM (#3793906)
You are absolutely correct that they are not mathematically identical. I struggled to word it in a way that would not mislead people, the distinction is important to emphasize.
Re:How Not To Sort By Average Rating by moultano on Tuesday April 03, 2012 @03:58PM (#3794534)
It has a really big effect I think on the tone of what gets selected at the top. The lower bound prefers things that are preferred by a majority and very popular. The prior method prefers things that are completely un-objectionable and liked by just enough people to be sure of that. My hunch is that with the lower bound you get more interesting things bubbling to the top because it puts a stronger emphasis on popularity.

In all of these models, the giant variable that is completely ignored is the actual choice to rate something at all, versus skipping over it and reading the next one. That's a very significant decision that the user makes. The behavior of each of these systems w.r.t that effect will be the dominant thing differentiating them.