Data Science Is America’s Hottest Job

Re:Data Science Is America’s Hottest Job by ska on Friday May 18, 2018 @01:59PM (#17103526)

You are input to a classification routine. The job of those designing the routine is to reduce the false positive rate to as close to 0 as they can without incurring too high a false negative rate.

So this sucks if you don't fit the model well in a way that has you often end up as a false negative - but that doesn't' mean the model is broken.

[ Parent | on HN ]

Re:Data Science Is America’s Hottest Job by kitanata on Friday May 18, 2018 @02:35PM (#17103822)

No. The model has a problem with overfitting. That’s my point.

[ Parent | on HN ]

Re:Data Science Is America’s Hottest Job by ska on Friday May 18, 2018 @08:03PM (#17105852)

Saying it doesn't make it true.

You are claiming there is a generalization problem that causes extra error in practice. Another perfectly viable hypothesis is that the classifier is working fine, it's just tuned for true positive rate and accepts a higher false negative rate to get it. Specificity vs. sensitivity is a fundamental trade off, not a training issue (though that can make both worse)

[ Parent | on HN ]