Re:A Look into Machine Learning's First Cheating Scandal
by masklinn on Monday December 14, 2015 @05:13PM
(#10734370)
There isn't an absolute limit on submissions, only a rate limit. The rate limit is probably somewhat arbitrary but it has a purpose: the aim of the challenge is to measure and compare algorithmic progress in machine learning, that's also why the test dataset is kept secret.
Without that, it becomes possible to pre-tune the algorithm to more closely match or better recognise the specific dataset ("hyperparameter optimisation") but not work any better in the general case, so the submitter does better on the specific challenge, but doesn't actually advance the field.
The limit is there to skew incentives towards algorithmic improvements, the specific rate doesn't really matter as long as it makes hyperparameter overfitting less convenient/efficient than algorithmic work.
The absolute number of submissions is somewhat arbitrary, e.g. why are 40 okay, but 200 are not?