Re:What scientific term or concept ought to be more widely k...
by Beldin on Wednesday December 08, 2021 @01:12AM
(#29481778)
Base Rate fallacy.
The gist: consider a yes/no test that has a small chance of being wrong (false positive and false negative). Examples: spam filter, virus filter, pregnancy test, testing for another illness, etc.
How well it performs in reality depends enormously on the real ratio of yes vs. no.
Example:
Spam filter is 99% correct, 1% false positive and equally 1% false negative. That sounds good! And it is - IF your spam vs ham rate is 50-50. If, however, 99% of mails are spam, and you use this filter on 10,000 mails, then:
- 9900 are spam; 100 ham
- 1% of ham = 1 mail ends up in spamfolder
- 1% of spam = 99 mails end up in inbox.
So your inbox now contains 99 ham mails and 99 spam - an unacceptable 50-50 ratio, even though your using a superduper spam filter.
Changing the base rate of spam affects the outcome drastically. Obviously, 99% ham and only 1% spam flips the numbers around: inbox is fine, but 50% of your spamfolder isn't spam. Also not really okay -- and that with exactly the same filter, just a different base rate.
TL;DR: actual effectiveness of tests which have false positives/negatives is heavily skewed by the frequency of occurrence of whatever you're testing for.
The gist: consider a yes/no test that has a small chance of being wrong (false positive and false negative). Examples: spam filter, virus filter, pregnancy test, testing for another illness, etc.
How well it performs in reality depends enormously on the real ratio of yes vs. no.
Example:
Spam filter is 99% correct, 1% false positive and equally 1% false negative. That sounds good! And it is - IF your spam vs ham rate is 50-50. If, however, 99% of mails are spam, and you use this filter on 10,000 mails, then:
- 9900 are spam; 100 ham
- 1% of ham = 1 mail ends up in spamfolder
- 1% of spam = 99 mails end up in inbox.
So your inbox now contains 99 ham mails and 99 spam - an unacceptable 50-50 ratio, even though your using a superduper spam filter.
Changing the base rate of spam affects the outcome drastically. Obviously, 99% ham and only 1% spam flips the numbers around: inbox is fine, but 50% of your spamfolder isn't spam. Also not really okay -- and that with exactly the same filter, just a different base rate.
TL;DR: actual effectiveness of tests which have false positives/negatives is heavily skewed by the frequency of occurrence of whatever you're testing for.