Re:A Look into Machine Learning's First Cheating Scandal
by fsam on Monday December 14, 2015 @12:51PM
(#10732521)
The fact that they created multiple logins was the biggest ethical breach here IMO. It was a willful violation of the contest rules.
The absolute number of submissions is somewhat arbitrary, e.g. why are 40 okay, but 200 are not?
Re:A Look into Machine Learning's First Cheating Scandal
by masklinn on Monday December 14, 2015 @05:13PM
(#10734370)
There isn't an absolute limit on submissions, only a rate limit. The rate limit is probably somewhat arbitrary but it has a purpose: the aim of the challenge is to measure and compare algorithmic progress in machine learning, that's also why the test dataset is kept secret.
Without that, it becomes possible to pre-tune the algorithm to more closely match or better recognise the specific dataset ("hyperparameter optimisation") but not work any better in the general case, so the submitter does better on the specific challenge, but doesn't actually advance the field.
The limit is there to skew incentives towards algorithmic improvements, the specific rate doesn't really matter as long as it makes hyperparameter overfitting less convenient/efficient than algorithmic work.
One of the article's notes indicates that it most likely does, and that the Baidu team wilfully got around that limit:
> Members of the Baidu team had to create multiple logins in order to circumvent the “two submissions per week” rule