Re:My experience with Kaggle's data-science competitions
by jbooth on Friday January 03, 2014 @10:55AM
(#7006020)
My problem with a lot of these competitions, particularly the high-reward ones, is you've got people duking it out at the 4th or 5th decimal point for the top few spots. At that point, you're not even competing for best generalizable algorithm, you're competing for 'algorithm which happens to fit the holdout set exceptionally well'. It's overfitting at a distance.
[ Parent
| on HN ]
Re:My experience with Kaggle's data-science competitions
by Houshalter on Friday January 03, 2014 @06:56PM
(#7009083)
The data set used for the leaderboard is not the dataset used in the final evaluation. They also limit the number of final submissions you can have and the number of submissions you can make per day to prevent that.
Often the person at the top of the leaderboard will move down a few places in the final evaluation.
Re:My experience with Kaggle's data-science competitions
by dhruvbhatia on Friday January 03, 2014 @08:49PM
(#7009527)
Further to Houshalter's point: the President & Chief Scientist of Kaggle addresses this specific issue and explains the Public/Private Leaderboard system in this webcast: http://youtu.be/GrugzF0-V3I