Re:AI competitions don’t produce useful models
by sgt101 on Friday September 20, 2019 @05:26AM
(#21024759)
If you look at the calculator and plug some values of your own you can see that the absolute value of the numbers is not what is generating the large sample sizes. The problem that is being shown up is the small gap in predictive power of the two classifiers being differentiated.
A common (industry) scenario is that you have a classifier that is 99.xx or 5x.xx accurate (it makes very few mistakes but they are costly, or it's a bit better than a coin flip, but we'll take that as it's what pays the mortgage), and we need to be absolutely positively certain that the one that we are fielding really works and is the best one we have (or homeless-a-go-go)
With the calculator
99.79 vs 99.65 @99% & 99% power-> 68457 examples
50.79 vs 50.65 @99 &99 -> 6129161 examples
which is why fancy models with marginal demonstrated improvement are often kept in the draw - much to the frustration of ML folks who are sure that it works and have proved that it will make the firm $10M a week.
A common (industry) scenario is that you have a classifier that is 99.xx or 5x.xx accurate (it makes very few mistakes but they are costly, or it's a bit better than a coin flip, but we'll take that as it's what pays the mortgage), and we need to be absolutely positively certain that the one that we are fielding really works and is the best one we have (or homeless-a-go-go)
With the calculator
99.79 vs 99.65 @99% & 99% power-> 68457 examples
50.79 vs 50.65 @99 &99 -> 6129161 examples
which is why fancy models with marginal demonstrated improvement are often kept in the draw - much to the frustration of ML folks who are sure that it works and have proved that it will make the firm $10M a week.