News for hackers. Stuff that matters.

A comment on BelKor has allegedly won Netflix Prize despite being second on the leaderboard. Parent comment

Re:BelKor has allegedly won Netflix Prize despite being seco... by jeresig on Monday July 27, 2009 @07:48AM (#725698)
And to be clear: The winner of the contest is not determined by the number that you see listed on the leaderboard. The number on the leaderboard [1] is on the Quiz dataset, the winner is determined by running against the Test dataset (which is kept private by Netflix). As stated by Yehuda on team BellKor: "our team is top contender for winning the Grand Prize, as we have a better Test score than The Ensemble."

This turnaround does not surprise me. BellKor's Pragmatic Chaos took their sweet time getting to 10%+ - and in doing so they were very sure not to overfit [2] the data (making their solution a much more generic and viable solution to the dataset). It's my guess that The Ensemble rushed quickly to 10%+ and overfit their data like mad (which yielded high numbers on the public dataset, but evidently does not translate to a generic solution).

I'm looking forward to seeing the final papers published by BellKor, et. al. - they're going to be a fascinating read, regardless.

1: http://www.netflixprize.com/leaderboard 2: http://en.wikipedia.org/wiki/Overfitting

[ Parent | on HN ]
Re:BelKor has allegedly won Netflix Prize despite being seco... by lliiffee on Monday July 27, 2009 @11:12AM (#725977)
I don't think that is correct. The numbers on the leaderboard are not the scores on the training set.

The netflix prize uses three datasets: the training set, the leaderboard set, and the test set. The training set is distributed to everyone. The test set is totally secret. Access to the leaderboard set is only by submitting results (once per 24 hours) and looking at the results. It is not at all trivial to "overfit" to this leaderboard set. (It could be done by, e.g. submitting results with slight tweaks to the algorithm parameters, but this would take a lot of time since you can only submit every 24 hours. Also, you would basically have to do it consciously.)

[ Parent | on HN ]
Re:BelKor has allegedly won Netflix Prize despite being seco... by jedc on Monday July 27, 2009 @08:10AM (#725717)
A lot of people have been tripped up by this, and it's a VERY important distinction. I hope Netflix posts more about the final (Test) results from each when they officially announce the winner.
[ Parent | on HN ]