News for hackers. Stuff that matters.
Re:Redefine statistical significance by BeetleB on Thursday September 07, 2017 @02:55PM (#15194730)
>I think the situation would improve with better teaching of philosophy of science and statistics (this would educate better reviewers too).

This is necessary, but not sufficient. What's needed is a way to know for sure that the hypothesis was not changed after data collection. I think predeclaring the hypothesis is the way to go.

Re:Redefine statistical significance by gattilorenz on Thursday September 07, 2017 @04:38PM (#15195485)
Yeah, but at the end you can still fabricate the data, remove "outliers",... Plus it's almost impossible to imagine a world where, before any experiment in any field, you predeclare it.

Not that education can fix all these (you can't prevent evil), but if reviewers and journals and conferences started to accept more the negative results, the incentive in lying would quickly decrease. And people would probably start to "disprove" interesting theories, instead of trying to "prove" niche results...

Re:Redefine statistical significance by BeetleB on Thursday September 07, 2017 @09:24PM (#15197243)
>Yeah, but at the end you can still fabricate the data, remove "outliers",...

Fabricating data is essentially fraud. And while it does happen, most of the problems with reproducibility are not problems of fraud.

It won't protect against outliers, but removing outliers will not solve most problems. It'll happen, but again, I don't think the majority of irreproducible studies are due to misuse of outliers.

>Plus it's almost impossible to imagine a world where, before any experiment in any field, you predeclare it.

Not at all. I'm not saying you predeclare every experiment - just every experiment you try to publish.

The way it works is:

1. You make observations (i.e. collect data - no predeclaring anything). If you see interesting patterns, you'll form a hypothesis.

2. This is the stage where you predeclare your hypothesis, and the criterion of falsification.

3. You now collect new data and test it against your hypothesis.

The hard part is ensuring people won't use some of the old data and claim they collected after their declaration. It's a hard problem, but not an impossible one.

People are in the habit these days of collecting a lot of data, seeing patterns, and publishing them. That's really not how a lot of early science was done. Once you see the patterns, you need to conduct more experiments to falsify them.

Re:Redefine statistical significance by setzer22 on Friday September 08, 2017 @05:40AM (#15198969)
Genuine curiosity here. What's wrong with making an experiment, and when the results clearly contradict your initial assumptions (so much that the opposite is confirmed) then publishing the found results?
Re:Redefine statistical significance by xaa on Friday September 08, 2017 @10:41AM (#15200494)
Because a very common type of hypothesis is along the lines of "gene X is important in disease Y" or "priming people with words will affect psychological outcome Z".

The opposite hypothesis is the null hypothesis which is "gene X is NOT important in disease Y" or "priming DOESN'T affect outcome Z".

Since we assume that most interventions will not affect most outcomes, these are much less surprising and interesting results. They are seen as "water is wet" type of findings and are thus hard to publish because no one is interested.

Now if your hypothesis is something like "X will cause Y to go up" and you actually find it causes Y to go down, that IS publishable. It is only when X has no effect on Y that you will have problems.

Re:Redefine statistical significance by BeetleB on Friday September 08, 2017 @11:06AM (#15200745)
If you mean in the context of what I was saying: Nothing wrong. But if you found some interesting correlations, you should not publish it - you should predeclare the hypothesis and what you'll look for, collect more data, and then publish if the new data matches the hypothesis.

This is assuming predeclaring ever becomes the norm.