Re:Gender Distribution in North Korean Posters
by anigbrowl on Monday October 16, 2017 @03:12PM
(#15485830)
I don't think extreme accuracy matters that much when dealing with large sample sizes. If the margin of error is under 3% (like opinion polls) then it's good enough for many purposes.
Re:Gender Distribution in North Korean Posters
by groceryheist on Monday October 16, 2017 @03:26PM
(#15485962)
The "margin of error" reported in opinion polls is typically just a function of the sample size and sample variance. The error is assumed to be due to statistical variation, not to problems with measurement. Measurement error is a bigger problem than statistical variation because statistical variation can often be assumed to be unbiased.
My comment above is not about statistical variation, it is about measurement error. Inaccuracies in models for labeling images are a form of measurement error. Having humans label a (often stratified) sample of images is required to understand the measurement error. This requirement is the main difficulty when it comes to generalizing learning from one context to another. If you use multiple independent humans to label all your images, you can understand the errors made by the humans very well.
Unfortunately, there is really no good reason for someone seriously interested in accurate information, like a researcher or journalist, to use machine learning for this particular task. Labeling a couple thousand images yourself or with friends is not that big of a task. Do it over a few evenings while watching TV and drinking beer. You could have mechanical turk workers do it for you for a few hundred dollars. In either case you will get extremely reliable information. If you use multiple judges you will have a good estimate of uncertainty for every classification. There is no way transfer learning can provide this uncertainty information.
The main advantage of this technique remains the ability to quickly label very large amounts of data on the order of hundreds of thousands of rows, or thousands of columns. For smaller data, machine learning can sometimes achieve marginal improvements in predictive performance through model complexity. However prediction in smaller data regimes is mainly useful for out-of-sample prediction. The machine learning paradigm offers limited support for measuring uncertainty for out of sample predictions, which is super important if you are a researcher.
One capability of transfer learning could be to support many many applications from one model, but I have yet to see demonstrations of this in practice. The problem is that knowing how well learning has transfered requires measuring generalizabity and so cannot be done blindly.