News for hackers. Stuff that matters.
Re:Redefine statistical significance by reilly3000 on Thursday September 07, 2017 @02:16PM (#15194398)
The cost of increasing sample size is significant; this is a trade-off that allows smaller projects to still conduct valuable research.
Re:Redefine statistical significance by Klockan on Thursday September 07, 2017 @02:47PM (#15194677)
Studies with low significance are no better than anecdotes.
Re:Redefine statistical significance by xelxebar on Thursday September 07, 2017 @07:38PM (#15196771)
Well, Bayes theorem should let us, in theory, combine several small studies into the equivalent of one larger one.

I'm practice it's harder than simple number crunching due to methodological differences between studies, but it seems a shame to call a study essentially worthless.

I like to be a little optimistic and think, "Data is data and data is good. The challenge is in interpreting it."

Re:Redefine statistical significance by _dps on Thursday September 07, 2017 @11:06PM (#15197625)
> "Data is data and data is good"

This is a reasonable intuition, but is hard to square with the increased risk of publication bias for small studies (i.e. we should question whether the typical small study published is really "data" in the sense we'd like).

Since small studies are more sensitive to noise and "fishing" for an effect, and since non-results are rarely published, what happens with meta-analysis of many published small studies is that you end up primarily looking at outliers rather than typical results. Since most research questions have a directionality to what counts as "interesting", the outliers also tend to be clustered on one side rather than evenly split.

The two papers I recommend anyone read if they care about these problems are:

Why Most Published Research Findings are False

http://journals.plos.org/plosmedicine/article?id=10.1371/jou...

The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time

http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

Re:Redefine statistical significance by Klockan on Friday September 08, 2017 @04:03PM (#15203493)
There is plenty of evidence that people who recreate results often tries until they get the same result as the original paper. For example a scientist made a mistake when counting the number of chromosomes and reported that humans had 48, a number that was replicated for over 30 years. In this case it would obviously have been better if the original paper was never published.

https://www.ncbi.nlm.nih.gov/pubmed/16463025

Re:Redefine statistical significance by csours on Thursday September 07, 2017 @05:02PM (#15195676)
https://xkcd.com/552/

Alt-Text: "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'."

I think you could replace correlation with "small studies" or "anecdotes". These are all things that suggest there may be some effect, and there may be merit in further study.

Re:Redefine statistical significance by Klockan on Friday September 08, 2017 @03:53PM (#15203432)
I agree that anecdotes are useful, I disagree with publishing them in scientific journals and telling the public that we found significant results.
Re:Redefine statistical significance by stdbrouw on Thursday September 07, 2017 @03:11PM (#15194834)
This is not true at all. Small studies are simply less accurate. Would you prefer to conduct further research on a cure for cancer that's been shown to lead to recovery in 0-50% of cases (the confidence interval includes 0%, so not significant) or in .5-1% of cases (significant)?
Re:Redefine statistical significance by imh on Thursday September 07, 2017 @10:47PM (#15197550)
As a matter of fact, I can do even better than 0-50% of cases! Consuming widgets has been shown to lead to recovery in 0-100% of cases! It isn't as simple as you suggest.
Re:Redefine statistical significance by stdbrouw on Friday September 08, 2017 @04:01AM (#15198626)
Yep, and that makes widgets a more interesting target for follow-up studies than a treatment that has been shown to result in no clinically significant improvement. It really is that simple.
Re:Redefine statistical significance by Klockan on Friday September 08, 2017 @03:50PM (#15203408)
Surely we could create a system to fund extensions of studies with promising initial results? I just think that it is dangerous to call 95% significant.
Re:Redefine statistical significance by Will_Parker on Thursday September 07, 2017 @07:09PM (#15196618)
How does it make sense to suppose it could lead to recovery in 50% of cases, if a drug that does nothing has a 5% chance of producing the same result in your study?
Re:Redefine statistical significance by stdbrouw on Friday September 08, 2017 @07:55AM (#15199529)
Because it has a 95% chance of producing a different (better) result. Statistics is all about the quantification of uncertainty.
Re:Redefine statistical significance by czep on Friday September 08, 2017 @08:15AM (#15199611)
That's true 95% of the time.
Re:Redefine statistical significance by Klockan on Friday September 08, 2017 @03:51PM (#15203416)
Do you believe that 95% of studies proving that humans possess paranormal abilities are true?