News for hackers. Stuff that matters.
Re:Understanding and explaining accuracy in machine learning... by sharkenstein on Wednesday March 13, 2019 @03:03PM (#19382311)
What are other metrics that can help validate the F1 score? I'm asking because I believe having a small sample size can skew the number or hide a flaw in the classification algorithm it's scoring.

In other words, what else should I ask to validate that a 70% F1-score is better than a 90% F1-score but on a smaller data set?

Re:Understanding and explaining accuracy in machine learning... by rsmith49 on Wednesday March 13, 2019 @03:20PM (#19382541)
That is a good point. It helps to think of Precision and Recall (so by extension F1-Score) from your test data as random variables sampled from a distribution modeling the probability of getting each value in your sample based on a "True" precision/recall value. I won't go too deep into the math, but this was part of the approach in the confidence calculations towards the end of the paper: being able to factor in the uncertainty of your classification metrics to confidence calculations.

To formally answer your question, the main things that matter in determining how stable your F1-Score from your test set is are: - Size of the test set - % of test set that has the label (in our case feedback tag) - the values found for precision and recall