What's "enough" depends entirely on what you're doing with it and how much accuracy you require. In this case, 86% was enough to improves their batting average and so was worth using.

If they were trying to detect pedestrians from a self-driving car then they'd need a lot more accuracy and so a lot more training data.

86% accurate should be a solid 7x speedup compared to manual selection, not to mention that the algo could also pre-populate description fields and only require a visual check-up instead of manual typing.

> Is it typical that ~2000 samples (1000 leads, 1000 non-leads) is "enough"?

In terms of practical machine learning, plotting learning curves are a great way to know if samples are "enough" [0, 1]. If your algorithm under fits (high bias) both errors (training and validation) will be high and in this case, adding training samples would not help.

[0] http://www.astroml.org/sklearn_tutorial/practical.html

[1] http://blog.fliptop.com/blog/2015/03/02/bias-variance-and-ov...

2000 samples is what we needed to get going yes. We're also noticing variable accuracy on different real world dataset. Meaning the algorithm is biased towards our current dataset.

Getting more data is one of our top priorities going forward with this project.

A follow-up question: In your experience, what's the cost difference between:

1) Mis-identifying a lead as a non-lead.

2) Mis-identifying a non-lead as a lead.

I would guess that (1) would be more costly if non-leads go to somewhere where they aren't followed-up on. But I'd appreciate the insight.

Thanks again!

Hi, as Per that wrote this is a developer while I do the sales that come out of this, I'll answer this:

Mis-identifying a lead as a non-lead is potentially loosing out on a big deal that can make or break your company. You never know what email will lead to a quickly closed $30k ARR sale, which are golden for any SaaS startup.

The reverse has almost no consequences unless you're really going to town with the emailing and end up being flagged for spam. Usually people just ignore you (not so smart) or write back that it's not relevant.

Your assumption is right, in terms of business cost, we'd rather see a false positive than a false negative.

I don't think the scikit learn algorithm differentiates between the two types of errors, in terms of cost.

Though it seems to give less false negatives than false positives overall, when testing on new datasets.

I'll put in the f1 score in the article when I have time.

Here's a whole post on that subject (or it might be Part 1 :)

http://blog.mldb.ai/blog/posts/2016/04/ml-meets-economics2/

This is exactly the link I was going to post. I think it does a great job showing that accuracy is not always the most appropriate metric for a model.

Statistical uncertainty grows like sqrt(N), and percent statistical uncertainty shrinks like sqrt(N) / N. So 2000 samples has a percent statistical uncertainty of about 2% - this is likely on par with or smaller than the systematic uncertainty (errors which don't depend on the size of the data). This means 2000 samples is probably fine for what they are trying to do, especially because the dataset is balanced with each class having 1000 samples. However the more data you have the better your model will do.

The author mentioned that 2000 was the minimum, and they intend to keep training the model with future data.