This conclusion assumes that running the test twice is equivalent to running two independent tests for the same condition. For many real tests, getting a false positive once predicts a higher chance of getting another false positive in a repeated test.
Time to bring out the pigeons.
Having just read the paper again, It’s fascinating.
In fact, "mammogram indeterminate" is a better diagnosis.
It's not, though.
Possible ≠ certain or even likely.
Screening tests are used to determine whether there are indications for more invasive but more conclusive diagnostic tests.
Yeah but "possible cancer", taken literally like that, is everyone.
This is the general argument against testing for prostate cancer in non-symptomatic patients. If you only test patients that has symptoms then the rate of false positives goes down. The downside is that you only catch late stage patients where treatment might be too late. The alternative (which is used for mammography) is that for every person saved, an order of magnitude more will be permanent harmed by unnecessary surgery.
It's also why breast cancer screening is not recommended for everyone, just women over 50.
For women over 50 there has been a decades long sincere effort by the medical community to take this all into account and the consensus is that screening makes sense in that population.
Prostate cancer being somewhat of an outlier as at least to my understanding, essentially all men will end up with some kind of tumor or another if they live long enough, but some large fraction of those won't become something that requires medical intervention before the individual would die from any number of other age-related causes. Outside my area of expertise but the relative size of the tissue that produces various breast cancers as well as the proximity to lymph nodes change the calculus on that a bit.
But I would definitely expect overall screening protocols to evolve as methodologies are developed that can potentially exclude the false positives from another metholodgy. While test A + A may still produce false positives due the nature of the test failing the same way, if we have A + B + C that are all relatively low-impact and not horribly expensive, that may end up being a fairly good program if some combination of A, B, and C generally exclude the parameters that the other produces a false positive for.
Except for the highly malignant variety prostate cancer is generally an "indolent" tumor, that is slow-growing. So in the majority of cases chances are good that the patient would die of something else before the tumor becomes the cause of death. However prostate cancer is a leading cause of cancer death in men. In the US ~30k/yr men die of prostate cancer, so it's not a trivial condition.
That sounds interesting. Why does lymph node proximity matter?
Almost all of them, in fact. If improving accuracy was as easy as repeating the test, the doctors would just be told that the correct procedure for this test is to take three readings.
You can expect that this is untrue only if the tests are especially expensive or invasive.
[0] https://en.wikipedia.org/wiki/Sensitivity_and_specificity [1] https://en.wikipedia.org/wiki/Positive_and_negative_predicti...
(edit yes of course cancer is not beneficial thank you)
I've read that's also a statistically tricky statement. IIRC, they measure survival time from when the cancer was detected. So early detection by definition improves the survival stats even when there's no meaningful treatment.
If I've read about this, I'm sure medical practitioners are well aware of it, but even so, I don't think it's easy to correct for it. How do you determine when a late-detected cancer actually started?
I am most familiar with NLST which actually was a long term randomized trial.
[edit: or, looking at the NLST you mentioned [1], maybe "treatment" doesn't always mean what I think of as treatment. They are actually comparing two different detection methods, and they aren't using a synthetic control.]
Do they ever use the "survival time after detection" stat I mentioned anymore, or has it been (/ should it be) abandoned?
Yes, NLST was actually a real randomized control trial for lung cancer screening, done before it was known whether or not it was helpful. The trial eventually stopped once they had enough outcome based evidence.
The statistic you mentioned is not used in assessing the effectiveness of screening. It does closely relate to what is though to be the causal mechanism that leads to effectiveness.
It's pretty uncontroversial that cancer screening for women above 50 years old or women with high risk (e.g. familial or genetic evidence or risk, history of cancer/treatment) is a net beneficial procedure.
For asymptomatic, low-risk women aged 40-49, the recommendations are a bit more mixed. While there is some evidence that overall mortality does improve, there are also considerable harms (physical, emotional, financial) associated with false positives and "overdiagnosis" (that is, identifying and treating low-grade non-invasive lesions that are unlikely to progress to breast cancer).
It would be interesting to know the mortality rate of unnecessary breast surgery. I would hope the number is less than 10%, but I suspect it is not an insignificant number of people. Net beneficial, but definitively controversial.
Living hapyy is more important than living for lots of people.
A breast cancer scare, is certainly harmful, but it's important to remember that it is going to be a short episode as it becomes clear that the tumor is not growing. So you have maybe a few months of scariness.
On the other hand cancer is brutal, and death is permanent. Years of life cut short. The years leading up that are extremely sad. The impacts of death extend beyond the person to their family and friends.
When people try their best to measure these things on balance, breast cancer screening is overall a good intervention.
Obviously with low-impact false positives like requiring more expensive or invasive, but not really dangerous, additional medical tests this is an easily accepted and accounted for set of outcomes, but as we try to do things like apply ML to ever more specific cases of complex questions it's worth keeping in mind.
Let's say two million people have been seriously inconvenienced by airport screening. We have to ask if that's an acceptable price to pay for the benefits? But do we know the benefits? Can we know the benefits? We actually don't know if the policy deterred anything, or even if someone who wanted to do something bad had their knife or liquid explosive or whatever confiscated and just abandoned the plan. It's purely a hypothetical so you're going to support the view you want: "we don't need to do all this because we don't actually have a big threat" or "we have successfully prevented any more attacks."
After all, we often see discussions here of foolish companies that underfund their security teams because "why are we paying so much for this when nobody has cyberattacked us?" ;)
When you have a condition that affects only 1% of the population, a test that reads negative 100% of the time will be 99% accurate.
A test that is always negative has 0% sensitivity and 100% specificity. Nobody thinks that sounds good.
Precision is not a good way to describe a test because, as I mentioned elsewhere, it relies on information about the population being tested.
So if you want to know "Has the Sun exploded?" you can flip a coin, assign heads to "Yes the Sun has exploded", and 50% of the time it will be correct.
You flip a coin, you get heads. You decide to flip the coin again, and you get heads again! What are the odds the Sun has exploded?
It's even worse in practice, given that the false positive rate is probably not independent between both measurements
Nitpick here. There's nothing wrong with the math of Bayesian probability, but the setup is usually something like an STD test, and in those cases the prior is wrong.
Our prior probability shouldn't be "0.01% (1 in 10,000) of the population is infected," we really want % of infection for the population that get tested.
This is really obvious with disease tests because people don't typically get tests without reasons (symptoms, contact with infected persons, etc.). The math as-is would only work if we're sampling people randomly.
Why am I forced to submit to these screening tests for issues I'm not at risk for in order to receive tangentially related medical care?
Current medical advice does NOT stipulate the two should be related - https://www.acog.org/womens-health/experts-and-stories/ask-a...
If you can't find a doc who won't decouple the two, try Planned Parenthood.
The correct answer is that the question doesn't give enough information to know.
I mean, a test that always said no would be 99.99% accurate. A mere 99% accuracy could be no false negatives and 0.99% of all tests false positives, in which case the answer is that a positive result is 1% likely to be correct, and 99% likely to be a false positive. Or it could be, on the other extreme, 0.01% false negative and 0.98% of all tests false positives, in which case the chance that a positive result represents an actual case would be 0%.
Presumably, though, approval processes for medical tests will constrain the actual distribution of results to be such that, a patient with positive result is not more likely to be clear of the condition than one with a negative result, so it won't be as bad as the latter extreme, but it's worth noting that your assumption (essentially that the false/true ratio is equal for negative and positive results) is not certain at all.
[1] https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
I remember reading studies about doctors not understanding the stats years ago, but my wife assures me that medical education has over corrected in the last decade or 2.
https://en.m.wikipedia.org/wiki/Sensitivity_and_specificity
> healthy people don't go in testing for the presense of a rare disease.
Well that’s precisely the point. There’s frequently the idea raised among lay people that it would be beneficial to test or screen for all kinds of rare or even just uncommon diseases that have high mortality in the general populace. This is the counter argument.
If the math is wrong, show us.
However, when it comes to mammograms, where the false negative rate is up to 12.5% and the false positive rate is up to 50%, and the incidence of breast cancer in women (12.5% over a lifetime) is rather high, false negatives would play more of a role.
The math is, given the test shows a positive, what is the probability that you actually have the disease?
If we are not making that assumption, then the 50% chance claim at the end is incorrect, since there's a chance that both of the two positive results are false positives and there is a false negative somewhere else in the remaining 99 (out of 101) people.
The 50% chance of it being a false positive (conditional on two positive results being observed) is actually a lower bound, and will increase as a function of the false negative rate.
First, I'm assuming that top comment meant the false-positive rate was 1%, even though it was poorly stated as "accuracy." If you're commenting on this original ambiguity, then the whole thing is ambiguous.
But if we accept a 1% false-positive rate, then it makes no difference at all what the false-negative rate is, once we have a positive test.
Out of a 100,000 tests, on average 1000 will be falsely positive. That's what a 1% false-positive means. It makes no difference how many are truly positive, how many are truly negative, and how many are falsely negative. The definition of 1% false-positive means that, out of 100,000 tests, on average 1000 will show a positive test when the condition isn't actually there.
So if 100,000 people each took two tests, on average 10 will have two false-positive tests. Again, this is independent of the number of false-negatives, and is simply the definition of false-positive. (Some tests may have correlated false-positives, i.e. you're more likely to get two in a row, but that's not standard and not part of the definition.)
So if you have a disease whose incidence is 10 people out of 100,000, and you took two tests which are both positive, you are equally-likely to be in the group of 10 people who really have the disease, and the group of 10 people sitting there with two positive tests who don't have the disease.
You seem to be saying that the percentages will be affected by whether any of the 10 out of 100,000 who really do have the disease got a negative test. But it doesn't. 1% false-positive means that, on average, 10 people who don't have the disease will get two false positive tests. It doesn't matter what the people who really have it get.
Positive predictive value is the likelihood that, if you have gotten a positive test result, you actually have the disease. It’s calculated as TP/TP+FP.
TP, ie true positives, is a function of false negatives.
Here's a common sense explanation. Assume the false negative rate is 100% and so you have zero true positives. Then every positive test result will be a false positive, and therefore a positive test result shows that you don't have the disease.
On the other hand, assume the false negative rate is 0. Then a positive test result will mean that you probably do have the disease as long as the false positive rate isn't too high.
Remember your post I was replying to was all about the probability that you actually have the disease conditional on a positive test result. That's the question I am addressing. For this question, you must consider the false negative rate.
A 1% false negative rate has a negligible impact on the math. 50% isn't perfect down to the decimal point, but it's very close.
After the first round you have 99.99 false positives and .99 true positives.
After the second round, if independent, you have .9999 false positives and .9801 true positives.
And in fact the lower bound, if you had no false negatives, is slightly below 50%. .9999 vs 1.
Their math shows that those are the same number, and they didn't elaborate on definitions.
The least misleading definition of accuracy is that accuracy = 1-false_positive_rate = 1-false_negative_rate. So unless told otherwise, I will use that definition.
"Let's say there is a terrible disease that 0.01% (1 in 10,000) of the population is infected with, and we develop a test that is 99% accurate. You go to the doctor and get a test. Oh no, it's positive. What are the chances that it's a false positive?
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100"
Then they conclude by saying:
"So there will be 100 false positives, and 1 real positive."
Or that they rounded .99 people to 1 person.
And given that they did round the false positives, I don't see why that's a strange idea.
Specifically, why would the 10,001st test carry a 50% chance of being a false positive?
Thus, the population for tests 1..10,000 is different than the population for tests 10,001..10,100.
Edit: I misunderstood the scenario.
If we test again, and the test is "fair", then of these 101 people, we will retest 99% (or 99) true negative, 1% (or 1) false positive, and 1 true positive. So two positives, one false positive and one true positive. 1 out of 2 is 50%.
(Tho, there is a chance also that the true positive might come back as a false negative, then 1 out of 1 or 100% of positive results would be false positive)
The silly thing with this demonstration is that accuracy is poorly defined. Tests are normally conditional on the actual presence of the disease or not.
Couldn’t it also alternatively be defined as 99% accurate for a given person?
Maybe there are reasons/conventions it is always as you originally describe, but if that’s the case wouldn’t it be more about poor communication to the public than a paradox?
As for why it is the case, I presume it is just the usual focus on counting your hits and not on your misses.
Now the conditional probability of infection, given TWO positive tests is still far below 50%.
This is why a second test of a different type, even if it is less accurate, if it is more likely to be independent, is much preferred!
https://github.com/khuyentran1401/Data-science/blob/master/s...
In a population of 1,000,000:
TP=99
TN=989901
FP=9999
FN=1
Accuracy = TN + TP / (TP+TN+FP+FN) = 99%
Sensitivity = TP / (TP + FN ) = 99%
Specificity = TN / (TN + FP ) = 99%
Looks like a great test across the board!
However, the actual chance someone with a positive test result has the disease:
TP / ( TP + FP ) = ~1%
So why isn't this ratio part of the major test criteria?
Because it has nothing to do with the test. Sensitivity and specificity are facts about the test, and conceptually they will remain whatever they are if you start testing a different population.
Whether doing a test has any value is a fact about the population ("olympic athletes are never anemic, so there's no point in testing them for anemia") that has nothing to do with the test. Why would you evaluate the test based on that?
I think it's because this number depends on the percent of the population that is positive which can change after the trial is conducted.
Is there a specific metric word or phrase for this less than 1% positive test result reliability?
Edit: Looks like hadmatter responded with the term I'm looking for!
Given that, from the first test, we have 100 people who were false positives, and 1 person who is truly positive. We would expect that 1 truly positive person to be positive again (we have a 99% chance of it from the test, after all), and 1 of the 100 who were false positive before to also test positive this second time (that 1% inaccuracy rate). Given 1 true positive, and 1 false positive, in that sample, you would expect a positive rate to mean you have a 50% chance of being the true positive.
If we did not have information from the first test, yes, it would be ~1% chance of a false positive. We have nearly a 0% chance of having a true positive, so of the 101 people, and a 99% accurate test, we would expect one to come out positive, despite not having the disease. But, because we have that first test, from a much larger sample size, we likely have one person who is a true positive, and so could expect two people to test positive, one with the disease, and one without (the 1/100 of the remaining), giving us a 50% chance a positive result = has the disease.
It's a Bayesian update. Every (independent) positive result increases your confidence of having the condition by 100x. At first you have no information so you assume the base rate, 1:10000. At the first test, that goes to 100:10000. Second test gets you to 10000:10000 or even odds. Third positive result finally gets you to 1000000:10000, or a 99% confidence of having the condition.
Let's say we've got a test that:
* If you've got cancer, detects it 100% of the time (to make the maths easy) * If you haven't got cancer, gives you a (true) negative 99% of the time, and a (false) positive 1% of the time.
For starters, consider a population of 100 people, all of whom have cancer. Our magical test always finds these cases, so we have an 100% positive rate, and a 0% false positive rate. (None of these can be false positives because we know they all really do have cancer.)
Now consider a group of 200 people, 50% of whom have cancer, and 50% of whom don't. The group who have cancer will still all get detected, but if the 100 people who don't have cancer, one of them will be marked as having cancer even though they don't. In total, that's 101 people who will get a positive result. So our overall positivity rate is 101/200 (~50%), but our false positivity rate is the number of false positives divided by the number of all positives, so 1/101 (~1%)
Now consider the case where we have 101 people, of whom only one has cancer. If we run our test a third time, we of course detect the person with cancer, but of the remaining 100 who don't have cancer, we will detect a second person. That's two positive cases in 101 people, so our positivity rate is ~2%. However, of those two positive cases we know only one really has cancer - the false positivity rate is 1/2, or 50%.
So that's three situations where the only thing we've varied is the proportion of the population who have the condition we're scanning for. (Technically we also changed how big the population was in the first place but that was just to make the maths easier - it doesn't affect the results at all.) However, we get a different rate of false positives in every test, because the number of false positives is dependent on the ratio between the number of people with cancer and the number of people without.
It's also worth noting that this obviously doesn't just apply to cancer, but also other situations. Most relevant to us in the last couple of years is probably Covid tests - you might have seen over the summers some scepticism about how valuable it is to do lots and lots of tests, and that's because during those times, the number of affected people was lower, and so these tests were more likely to produce less useful results. However, in winter, even relatively poor tests become more useful, because there are more true positives, and therefore a lower rate of false positives.
Intuition would tell you 1% since the test is 99% accurate. Intuition, as it often is, is wrong. Take a sample size of 'x', we'll use 10,000. How many people will be infected? 1. How many false positives will there be? 1% * 10,000 = 100. So there will be 100 false positives, and 1 real positive. So it turns out there's a greater than 99% chance that your result is a false positive, even though the test is genuinely 99% accurate.
Here's a fun one. So we decide to run the test twice. And AGAIN it comes up positive. What are the odds that it's a false positive? Again, the exact same logic holds - there are 101 people we are testing. There will be about 1 false positive and about 1 real infection. So the odds of it being a false positive once again are about 50%.