I agree the lower bound of a test statistic is a pretty indirect way of encoding intuitions. Somehow I tend to find loss functions the conceptually clearest way of encoding preferences about inference outcomes, though. But, in this case my feeling in that direction isn't very strong, and the priors-based solution seems fine.
There's a distinct difference in the asymptotic behavior though between the lower bound and the prior. The lower bound goes to the mean as 1/sqrt(n), the prior goes to the mean as 1/n.
That makes for a pretty significant difference in practice, and I'm not sure which is preferable.
Re:How Not To Sort By Average Rating
by NathanRice on Tuesday April 03, 2012 @01:30PM
(#3793906)
You are absolutely correct that they are not mathematically identical. I struggled to word it in a way that would not mislead people, the distinction is important to emphasize.
It has a really big effect I think on the tone of what gets selected at the top. The lower bound prefers things that are preferred by a majority and very popular. The prior method prefers things that are completely un-objectionable and liked by just enough people to be sure of that. My hunch is that with the lower bound you get more interesting things bubbling to the top because it puts a stronger emphasis on popularity.
In all of these models, the giant variable that is completely ignored is the actual choice to rate something at all, versus skipping over it and reading the next one. That's a very significant decision that the user makes. The behavior of each of these systems w.r.t that effect will be the dominant thing differentiating them.
The strength of priors here is that it is very easy to take intuitions and encode them statistically, in an understandable way. Taking the lower bound of a test statistic doesn't admit much in the way of intuition.