I do love how counterintuitive sample sizes are. I swear I see the "it's intuitively obvious that 45 people is a comically small sample size, the researcher is an idiot and this is useless!" comments all the time. Then somebody does the math and it turns out p ≈ 0

Yeah... and then there's the people who benchmark computer programs, don't even bother measuring the test variance, don't know about warming the cache, and post raw timing data online claiming that X is better than Y. Dunning Kruger is a harsh mistress.

Let's apply Bayesian reasoning. :P

P(skill): Let's choose a prior that someone can smell parkinson's as one in a million, or 10^-6. (Not very well argued, I admit.)

P(data): This exact data's random occurrence probability is 1/2538.

P(data|skill): The probability of the result, taking account that she has the skill, is 1 (this assumes she never errs).

So we get

    P(skill|data) = P(data|skill) x P(skill) / P(data)

    P(skill) = 1 x 10^-6 / (1/2538) = 0.002538

Or 0.25 percent probability, based on this test, that she has the skill. Which is low.

Intuitively, I would have expected the calculation to yield a much higher number. The prior was very low though.

So I think the grandparent post has some merit. It can be argued that the claim is so extraordinary (the prior) that even twelve "coin tosses" guessed right in a row is more likely.

I think your probabilities are way off. We know that some people can smell some disease factors already (fetor hepaticus). Additionally, there's a LOT of genetic smell differences -- cilantro, cucumber, asparagus, cyanide, etc. Knowing these facts drastically affects both the P(smell) and P(skill) (which you've combined into P(skill)).

Additionally, the likelihood that a particular person would be so convinced in their ability to smell Parkinson's that they sufficiently convince a few researchers AND THEN predict with perfect accuracy is much smaller than 1/4096. This isn't a random person picked off the street--this is a person specifically claiming to be able to accomplish a feat and then being successful on the first try at 1/4096 odds (1/2^12).

Combined, I find this result very interesting and strongly believe the validity of this research. Granted, just because something is likely doesn't necessarily make it true, but it's certainly a strong impetus for future research, or even for people with a similar talent to come forward.

There's a missing variable here: she claimed that she can tell the difference.

We are interested in the following calculation:

    P(skill|claim&data) = P(claim&data|skill) x P(skill) / P(claim&data)

The upper limit of P(claim&data) is P(claim). If P(claim) were above 10^-4, I think the researchers would have met somebody else with the claim, so 10^-4 is a reasonable pessimistic value.

So, Bayesian reasoning won't save you here.

You're exactly right, if the researchers picked a random person off the street and made them smell shirts. But they didn't.

How many people claim they can smell Parkinson's? How many otherwise truthful, sane people have falsely made this claim? Perhaps a better approach to a prior would take into consideration that she claims that she can smell Parkinson's and her reputation as a truthful, sane person. My prior wouldn't be less that 0.1.

That's a good point. On the other hand, if it was common for people (one in a thousand) to smell Parkinson's, somebody would have come forward 200 years ago? A doctor or nurse, perhaps. So, on population level it should be extremely rare.

Of course, she is not a person picked at random, we should look at the population of people claiming to have done these sort of things already on their own. This population would be expected to contain a large portion of people with mental issues and charlatans trying to gain some financial benefit. If she doesn't have a history of either, then the prior jumps up to a very high level already. It is not very likely for a normal person to claim this sort of thing, unless they already have good evidence by themselves already.

So, with no scientific basis at all you believe there is a 10% chance that people can detect disease by smell, something that has AFAIK never been proven for any disease? We don't go to the doctor and have him or her smell our armpits. Do you use a psychic? What we have here is an organization that is basing their research funding on math tricks that I, ten years out of undergrad, maybe didn't remember exactly as it was characterized but was able to see through.

There are so many things wrong in this comment it's hard to figure out where to begin.

"We don't go to the doctor and have him or her smell our armpits" is an argument to authority. Just because a doctor doesn't use test X does not mean that test X is not useful. Every single diagnosis test we use today was, at some point in the past, unknown and unused by doctors. It is scientific research which gave us those tests. And, because you seem to be uninformed about the subject, I'd like to tell you that there are a number of things that a doctor will smell when they diagnose you. Famously, you can diagnose phenylketonuria by smell, and you can also diagnose diabetes by smell.

This comment also seems to reflect a fundamental misunderstanding of the scientific process. The whole point of scientific research--which requires funding, usually--is to figure out if a hypothesis is true or false. If you already know whether your hypothesis is true or false, you're not doing research, you're replicating results.

When you do preliminary research, it's because you don't have very good information about some particular subject. You're complaining about the shaky ground that they base their research funding on--but these scientists did the right thing. Because the hypothesis seemed improbable, they conducted a dirt cheap experiment. It's an experiment that you could have conducted yourself for $20.

> If you already know whether your hypothesis is true or false, you're not doing research

That's exactly what this is. People believe the claim 100% so we do a simple coin flip, and, yes, there it is. It's confirmed! No extraordinary evidence required for this extraordinary claim. Let the research money flow and the BBC reporting commence. If it were my money I would have another lab repeat the experiment.

The prior in this case isn't for people in general, it is for this person.

You may want to revisit your stats, since there's no "math tricks" here. Even if you assume she knew that some sizable portion of the shirts came from AD patients, the p-val is quite low. If you assume every shirt was independent, so there's no knowledge to gained by knowing how many other shirts she classified as AD, the p-val is ludicrously low. And if you go Bayesian, no matter how you slice the prior probabilities, it's still low.

First, what p-value are you referring to? You aren't even stating what your test is. Please explain how you got to "p-value is quite low"

Look at the various analyses in the threads. Mine was a frequentist analysis based on independent categorizations, which has a p-val of ~.0002. Others have posted more sophisticated frequentist and Bayesian analyses based on priors and the subject having advance knowledge of the number of ADs present.

But no matter what assumptions were made, no p-val was greater than .001, which is quite low for n=12 with a single test. Our generally accepted threshold is p<.05. She literally had a perfect score.

Also, saying "An actual test would need to allow any possible sample including those that had zero Parkinson's patients" indicates you don't understand experimental design. Splitting the data into equal groups maximizes your chances of detecting something when effect sizes are small, since sensitivity is related to minimum group size. (P-values are hurt more by low sample sizes than they gain by large ones, which is why a 3/9 split is less powerful than a 6/6 split.)

Not only did she have a perfect score, but she "adamantly" corrected a mistake in the control group. That must have some impact on the Bayesian estimate too. Her correction is worth something, and her confidence in providing that correction is worth something.