Re:AI competitions don’t produce useful models
by fishooter on Friday September 20, 2019 @02:56AM
(#21023883)
The post talks about coin flipping, or 0/1 classification. Many competitions use different scores however - multiclasses, learning to find bounding boxes of objects, etc. It is much less likely to find "good" answers on the test set by chance. I think the points in the article are important, but with this context become a non-issue, when a random answer is unlikely to be correct.
[ Parent
| on HN ]
Re:AI competitions don’t produce useful models
by MrMoenty on Friday September 20, 2019 @03:12AM
(#21023967)
The article is not about models being indistinguishable from random classifiers, the difference there should be very significant even on the tasks it discussed. Instead, the problem originates from the small differences in test set performance between the top N models. While that difference may very well increase when moving from binary classification to a more technically involved regression task, that is by no means guaranteed, and the main points of the article still apply.