You're right... you are too lazy.

Traditional null hypothesis would be something like "she guesses right 50% of the time", which gives a likelihood of 1/16384 that she would get the correct answer. Let's be cynical, and suppose that she knew or guessed that there were 5-7 patients with Parkinson's, the likelihood is now 1/2538, still pretty low. Even if she knew there were exactly 7 patients with Parkinson's (quite a cynical null hypothesis!) the likelihood is only 1/792. Hey, that's a p-value of 0.0012!

Yes, p-values suck. But, the significance is absolutely there, but we would want to follow this research up with a larger sample size and control more of the variables.

I do love how counterintuitive sample sizes are. I swear I see the "it's intuitively obvious that 45 people is a comically small sample size, the researcher is an idiot and this is useless!" comments all the time. Then somebody does the math and it turns out p ≈ 0

Yeah... and then there's the people who benchmark computer programs, don't even bother measuring the test variance, don't know about warming the cache, and post raw timing data online claiming that X is better than Y. Dunning Kruger is a harsh mistress.

Let's apply Bayesian reasoning. :P

P(skill): Let's choose a prior that someone can smell parkinson's as one in a million, or 10^-6. (Not very well argued, I admit.)

P(data): This exact data's random occurrence probability is 1/2538.

P(data|skill): The probability of the result, taking account that she has the skill, is 1 (this assumes she never errs).

So we get

    P(skill|data) = P(data|skill) x P(skill) / P(data)

    P(skill) = 1 x 10^-6 / (1/2538) = 0.002538

Or 0.25 percent probability, based on this test, that she has the skill. Which is low.

Intuitively, I would have expected the calculation to yield a much higher number. The prior was very low though.

So I think the grandparent post has some merit. It can be argued that the claim is so extraordinary (the prior) that even twelve "coin tosses" guessed right in a row is more likely.

I think your probabilities are way off. We know that some people can smell some disease factors already (fetor hepaticus). Additionally, there's a LOT of genetic smell differences -- cilantro, cucumber, asparagus, cyanide, etc. Knowing these facts drastically affects both the P(smell) and P(skill) (which you've combined into P(skill)).

Additionally, the likelihood that a particular person would be so convinced in their ability to smell Parkinson's that they sufficiently convince a few researchers AND THEN predict with perfect accuracy is much smaller than 1/4096. This isn't a random person picked off the street--this is a person specifically claiming to be able to accomplish a feat and then being successful on the first try at 1/4096 odds (1/2^12).

Combined, I find this result very interesting and strongly believe the validity of this research. Granted, just because something is likely doesn't necessarily make it true, but it's certainly a strong impetus for future research, or even for people with a similar talent to come forward.

There's a missing variable here: she claimed that she can tell the difference.

We are interested in the following calculation:

    P(skill|claim&data) = P(claim&data|skill) x P(skill) / P(claim&data)

The upper limit of P(claim&data) is P(claim). If P(claim) were above 10^-4, I think the researchers would have met somebody else with the claim, so 10^-4 is a reasonable pessimistic value.

So, Bayesian reasoning won't save you here.

You're exactly right, if the researchers picked a random person off the street and made them smell shirts. But they didn't.

How many people claim they can smell Parkinson's? How many otherwise truthful, sane people have falsely made this claim? Perhaps a better approach to a prior would take into consideration that she claims that she can smell Parkinson's and her reputation as a truthful, sane person. My prior wouldn't be less that 0.1.

That's a good point. On the other hand, if it was common for people (one in a thousand) to smell Parkinson's, somebody would have come forward 200 years ago? A doctor or nurse, perhaps. So, on population level it should be extremely rare.

Of course, she is not a person picked at random, we should look at the population of people claiming to have done these sort of things already on their own. This population would be expected to contain a large portion of people with mental issues and charlatans trying to gain some financial benefit. If she doesn't have a history of either, then the prior jumps up to a very high level already. It is not very likely for a normal person to claim this sort of thing, unless they already have good evidence by themselves already.

So, with no scientific basis at all you believe there is a 10% chance that people can detect disease by smell, something that has AFAIK never been proven for any disease? We don't go to the doctor and have him or her smell our armpits. Do you use a psychic? What we have here is an organization that is basing their research funding on math tricks that I, ten years out of undergrad, maybe didn't remember exactly as it was characterized but was able to see through.

There are so many things wrong in this comment it's hard to figure out where to begin.

"We don't go to the doctor and have him or her smell our armpits" is an argument to authority. Just because a doctor doesn't use test X does not mean that test X is not useful. Every single diagnosis test we use today was, at some point in the past, unknown and unused by doctors. It is scientific research which gave us those tests. And, because you seem to be uninformed about the subject, I'd like to tell you that there are a number of things that a doctor will smell when they diagnose you. Famously, you can diagnose phenylketonuria by smell, and you can also diagnose diabetes by smell.

This comment also seems to reflect a fundamental misunderstanding of the scientific process. The whole point of scientific research--which requires funding, usually--is to figure out if a hypothesis is true or false. If you already know whether your hypothesis is true or false, you're not doing research, you're replicating results.

When you do preliminary research, it's because you don't have very good information about some particular subject. You're complaining about the shaky ground that they base their research funding on--but these scientists did the right thing. Because the hypothesis seemed improbable, they conducted a dirt cheap experiment. It's an experiment that you could have conducted yourself for $20.

> If you already know whether your hypothesis is true or false, you're not doing research

That's exactly what this is. People believe the claim 100% so we do a simple coin flip, and, yes, there it is. It's confirmed! No extraordinary evidence required for this extraordinary claim. Let the research money flow and the BBC reporting commence. If it were my money I would have another lab repeat the experiment.

The prior in this case isn't for people in general, it is for this person.

You may want to revisit your stats, since there's no "math tricks" here. Even if you assume she knew that some sizable portion of the shirts came from AD patients, the p-val is quite low. If you assume every shirt was independent, so there's no knowledge to gained by knowing how many other shirts she classified as AD, the p-val is ludicrously low. And if you go Bayesian, no matter how you slice the prior probabilities, it's still low.

First, what p-value are you referring to? You aren't even stating what your test is. Please explain how you got to "p-value is quite low"

Look at the various analyses in the threads. Mine was a frequentist analysis based on independent categorizations, which has a p-val of ~.0002. Others have posted more sophisticated frequentist and Bayesian analyses based on priors and the subject having advance knowledge of the number of ADs present.

But no matter what assumptions were made, no p-val was greater than .001, which is quite low for n=12 with a single test. Our generally accepted threshold is p<.05. She literally had a perfect score.

Also, saying "An actual test would need to allow any possible sample including those that had zero Parkinson's patients" indicates you don't understand experimental design. Splitting the data into equal groups maximizes your chances of detecting something when effect sizes are small, since sensitivity is related to minimum group size. (P-values are hurt more by low sample sizes than they gain by large ones, which is why a 3/9 split is less powerful than a 6/6 split.)

Not only did she have a perfect score, but she "adamantly" corrected a mistake in the control group. That must have some impact on the Bayesian estimate too. Her correction is worth something, and her confidence in providing that correction is worth something.

Well, to be conservative, assume she was told or guessed that 7 out of the 12 samples were from patients with Parkinson's, which more than accounts for the "predisposed to thinking at least some" factor - actually it would make more sense to guess six, but that can only make the correct response less likely. Twelve choose seven is 792, so out of the 792 possible responses, she chose the single exactly correct one. No fancy statistics needed - the p-value (defined, according to Wikipedia, as the probability of obtaining an equal or more extreme result, the latter of which doesn't apply as the result is already the most extreme possible for the experiment) is simply 1/792 or about 0.001 (0.1%).

That's ridiculous. That's like saying someone who wins the lottery is clairvoyant with P=0.00. I mean they got the exact right numbers which had 1 in 147 million chance, right?

A lot of people play the lottery. Only one person did this experiment. When you test multiple hypothesis (as in your lottery example), you need to perform a correction[2,3].

[1] https://xkcd.com/882/

[2] https://en.wikipedia.org/wiki/Multiple_comparisons_problem

[3] https://en.wikipedia.org/wiki/Bonferroni_correction

That's totally different. They didn't test a whole bunch of people a whole bunch of times. That's what the lottery is. That's different math.

How do I know that? I just assume the guy that won was the only guy that played. If we can forget about all the people who ever claimed to smell disease, then we can forget about all the other lotto players.

What I find amusing here is the only intellectually honest way you have out of the logic hole you've dug is to claim that you understand neither the lottery, nor the article, which then makes your opinion on either uninformed and not terribly compelling.

Were you just advocating for the devil?

I believe there are some treatments out there aimed at slowing down the progress of the disease. Diagnosing early might mean slowing down the nasty stuff earlier, thus improving the life of the patient. Also, this sounds like something that is not at all part of the known symptoms of disease. Imagine if the change of smell is linked to something that is a cause for parkinson (I'm not suggesting the article even considers that option, but it's not impossible), and imagine that that something is actually "easy" to treat for. That would change a lot of things ! Yes, it's very unlikely, but even a small increase in the knowledge of a disease is always progress.

Here is a table relating the sample size, sensitivity, and confidence interval for evaluating diagnostic medical tests http://www.nature.com/nrmicro/journal/v8/n12_supp/fig_tab/nr...

As you can see, a sensitivity of 95% (true positive 95% of the time) with a confidence interval +- 4.3% requires 100 positive subjects. Because the natural rate is 1 in 500, we would need 500*100 = 50,000 total subjects. So you can see how absolutely ludicrous it is to say a woman sniffs the clothes of 12 subjects and is presumed to have a 100% true positive rate with 100% confidence level.

You clearly need to take stats again, because you don't understand what you're citing. A 95% confidence interval is not the same thing as a hypothesis test of p<.05. A confidence interval is for estimating the uncertainty that your chosen margin of error will include the true population parameter.

Nobody's claiming she's always 100% accurate. That's also a factor of low sample sizes. But I went ahead and computed the margin of error for you. For n=12, a 95% confidence interval requires a margin of error of 28%, so her true detection ability is, at worst, 72%, which is still higher than anything else we've got.

The odds of randomly guessing all 12 correctly is 1 in 2^12, or 1 in 4096.

This assumes that people 50% of people have Parkinson's and that the people are chosen randomly from the population. Unfortunately neither of those things are true.

The odds of randomly drawing a royal flush are 1 in 649,000 but it happens all the time.

That's because there are hundreds of thousands or millions of draws occurring. You need to account for what's called the multiple testing problem. [1] Some ways to do this are to control the family-wise error rate (FWER) or the false discovery rate (FDR). You can control the FWER with the Bonferroni correction or the Sidak correction and you can control the FDR with the Benjamini-Hochberg procedure. You are completely correct about things like this happening by chance, and this is the cause of publication bias in science (since there is a predisposition for publishing positive results).

1. https://en.wikipedia.org/wiki/Multiple_comparisons_problem

No, it doesn't happen all the time. It happens, on average, 1 time in 649,000. That's what those odds mean.