News for hackers. Stuff that matters.
Re:Redefine statistical significance by gattilorenz on Thursday September 07, 2017 @12:19PM (#15193209)
It can hardly hurt, but it is still a stop gap measure. It won't solve the publication bias people will still change the hypothesis or the test after measurements are done.

I think the situation would improve with better teaching of philosophy of science and statistics (this would educate better reviewers too).

Re:Redefine statistical significance by agentofoblivion on Thursday September 07, 2017 @12:27PM (#15193320)
Agree. It doesn't stop p-hacking, it just makes it harder. Definitely treating the symptom instead of the disease. We ultimately need institutional and cultural change, but it's not obvious how to do that in the short term, so making it harder to claim significance might be a step in the right direction.

On the other hand, you might expect that new discoveries by nature have less data since data is likely more expensive for brand new research, and by extension a lower likelihood of meeting these sorts of stringent statistical requirements. Decreasing the p-value threshold may be counter-productive if we dismiss legitimate new discoveries due to essentially economic constraints with data gathering, which would have the impact of making it less likely to get funding to pursue the problem in more depth, thereby slowing the advance of discoveries.

Re:Redefine statistical significance by jjoonathan on Thursday September 07, 2017 @01:02PM (#15193719)
> It doesn't stop p-hacking, it just makes it harder.

I could see the reverse happening, where higher p-value standards lead to normalization of deviance in the form of worse p-hacking.

Re:Redefine statistical significance by felippee on Thursday September 07, 2017 @01:27PM (#15193942)
I agree with that. In my prediction legitimate, honest research with p~0.05 will become unpublishable, while p-hacked bullshit will prevail in glory.
Re:Redefine statistical significance by PeterisP on Thursday September 07, 2017 @04:39PM (#15195496)
If you do a honest preliminary study and get a p~0.05 result, then repeating the study with 70% larger sample size should get you p~0.005; but if you've p-hacked, it won't.
Re:Redefine statistical significance by felippee on Thursday September 07, 2017 @04:58PM (#15195648)
But sometimes getting 70% larger sample will cost you so much money, that you'd need another grant, one you'd get after you'd publish your p=0.05 result, which given the new rules you'd not be able to publish...

Statistics is just another method and almost every method can be hacked or abused. Science is not about putting checkmarks in tables but about reading and understanding ideas and reproducing the results. Tweaking some numeric values is not going to help the review process which is fundamentally broken these days.

Re:Redefine statistical significance by zeckalpha on Friday September 08, 2017 @05:19AM (#15198891)
This is addressed in the paper. The additional costs of larger sample sizes outweigh the costs of studies based on false positives.
Re:Redefine statistical significance by epistasis on Thursday September 07, 2017 @01:51PM (#15194143)
It can hurt, in that it can slow the spread of information. If you perform 70% fewer different types of experiments because you have to hit p=0.005 instead of p=0.05, then you explore in fewer directions.

This is a classic tradeoff between exploration and exploitation in active learning.

If your view of the world is that there are only a very few hypotheses worth exploring, and you have a good lay of the scientific land, then requiring higher bar of proof is probably good.

If it's a new field that's extremely complex and where very little is known of the governing principles, then requiring very high stats could severely slow progress and waste lots of research dollars.

I completely agree that rather than setting arbitrary barriers for significance, it would seem much better to let people actually understand what was found, at whatever significance it was. Even setting up the null model to get a p-value requires tons and assumptions. The better test is reproducibility and predictive models that can be validated or invalidated. That's where the science is, and not in the p.

Re:Redefine statistical significance by nonbel on Thursday September 07, 2017 @01:58PM (#15194218)
> "It can hurt, in that it can slow the spread of information."

I am not at all in favor of this proposal, but one thing it may do is stem the tidal wave of misinformation.

Re:Redefine statistical significance by btilly on Thursday September 07, 2017 @03:13PM (#15194854)
Yours is a theoretical concern.

The very practical concern is that entire areas of research have been based on studies replicated and backed up entirely through p-hacking and selectively publishing only papers with positive results. This is a proven issue today. See https://en.wikipedia.org/wiki/Replication_crisis for more.

It may be that there is a pendulum that needs to swing a few times to get to a good tradeoff. But it is clear, now, which direction it needs to swing.

Re:Redefine statistical significance by epistasis on Thursday September 07, 2017 @05:04PM (#15195696)
I disagree 100%, having read that Wikipedia page and its sources.

It's something that affects a few fields, not all science. And the problem has been completely 100% overblown.

If the problem is that things aren't replicating, changing the p-value cutoff for significance isn't going to fix everything. It can just as easily be a bad null model that's the problem,in which case you can't trust any p-value. The MRI scan problem was closely related to that.

It's a field-specific and null-model specific thing. Broadly changing the a p-value cutoff for everybody isn't going to fix this issue.

Re:Redefine statistical significance by nonbel on Friday September 08, 2017 @08:01AM (#15199552)
>"And the problem has been completely 100% overblown"

Just the fact that so few replications have been published indicates huge cultural problems. When I did biomed, in my tiny area of expertise there had been 1-2 thousand papers published since the 1980s. Out of these maybe 2-3 were close to direct replications. None of those showed the results were reproducible, but no one cared...

Usually there were "minor differences" in the methods so it resulted in stuff like: "Protein P has effect E by acting through receptor R in cell line L from animal A of sex S and age Y when in media M".

However if you changed L, A, S, Y, or M apparently totally different things were going on (there were then supposedly dozens of receptors for each ligand, each receptor having dozens of ligands in different circumstances, etc).

In the end I found that E was nearly perfectly correlated with the molecular weight of P (using data from one of the most cited papers on the topic, in which they specifically claimed there was no correlation with any physical properties of the ligands).

So the effect has nothing to do with specific ligand-receptor interactions at all, but no one cared. Situations like this (with few published direct replications, the ones that are published are contradictory, the results are all being misinterpreted anyway, and everyone just continues on their way when problems are pointed out) are totally standard for biomed. The replication aspect of the issue is really only the tip of the iceberg of problems.

Re:Redefine statistical significance by gboudrias on Thursday September 07, 2017 @04:51PM (#15195596)
> It won't solve the publication bias people will still change the hypothesis or the test after measurements are done.

As a Psychology student, this is a well-known initiative: https://cos.io/prereg/

(Though I can't confirm or deny its widespread usage.)

The publication bias is harder, and pre-registration won't solve this. But I think this is a separate issue, and it's important to address each issue in its own right.

I've seen the proposal from TFA before and with my very limited knowledge, I'm still fairly certain it will never come to pass in Psychology, as nearly half of all modern studies have reproducibility issues (!). It would be beneficial to our field, in the way that a band-aid is beneficial to a gaping wound, but it would require a lot more rigor than has been evidently been displayed so far (and more rigor is more work, and time is limited).

So... Don't hold your breath.

(Sorry if my comment sounds pessimistic, I don't know much and I'm open to being corrected. I still have enough critical thought to be skeptical of some researchers' dedication to intellectual rigor.)

Re:Redefine statistical significance by BeetleB on Thursday September 07, 2017 @02:55PM (#15194730)
>I think the situation would improve with better teaching of philosophy of science and statistics (this would educate better reviewers too).

This is necessary, but not sufficient. What's needed is a way to know for sure that the hypothesis was not changed after data collection. I think predeclaring the hypothesis is the way to go.

Re:Redefine statistical significance by gattilorenz on Thursday September 07, 2017 @04:38PM (#15195485)
Yeah, but at the end you can still fabricate the data, remove "outliers",... Plus it's almost impossible to imagine a world where, before any experiment in any field, you predeclare it.

Not that education can fix all these (you can't prevent evil), but if reviewers and journals and conferences started to accept more the negative results, the incentive in lying would quickly decrease. And people would probably start to "disprove" interesting theories, instead of trying to "prove" niche results...

Re:Redefine statistical significance by BeetleB on Thursday September 07, 2017 @09:24PM (#15197243)
>Yeah, but at the end you can still fabricate the data, remove "outliers",...

Fabricating data is essentially fraud. And while it does happen, most of the problems with reproducibility are not problems of fraud.

It won't protect against outliers, but removing outliers will not solve most problems. It'll happen, but again, I don't think the majority of irreproducible studies are due to misuse of outliers.

>Plus it's almost impossible to imagine a world where, before any experiment in any field, you predeclare it.

Not at all. I'm not saying you predeclare every experiment - just every experiment you try to publish.

The way it works is:

1. You make observations (i.e. collect data - no predeclaring anything). If you see interesting patterns, you'll form a hypothesis.

2. This is the stage where you predeclare your hypothesis, and the criterion of falsification.

3. You now collect new data and test it against your hypothesis.

The hard part is ensuring people won't use some of the old data and claim they collected after their declaration. It's a hard problem, but not an impossible one.

People are in the habit these days of collecting a lot of data, seeing patterns, and publishing them. That's really not how a lot of early science was done. Once you see the patterns, you need to conduct more experiments to falsify them.

Re:Redefine statistical significance by setzer22 on Friday September 08, 2017 @05:40AM (#15198969)
Genuine curiosity here. What's wrong with making an experiment, and when the results clearly contradict your initial assumptions (so much that the opposite is confirmed) then publishing the found results?
Re:Redefine statistical significance by xaa on Friday September 08, 2017 @10:41AM (#15200494)
Because a very common type of hypothesis is along the lines of "gene X is important in disease Y" or "priming people with words will affect psychological outcome Z".

The opposite hypothesis is the null hypothesis which is "gene X is NOT important in disease Y" or "priming DOESN'T affect outcome Z".

Since we assume that most interventions will not affect most outcomes, these are much less surprising and interesting results. They are seen as "water is wet" type of findings and are thus hard to publish because no one is interested.

Now if your hypothesis is something like "X will cause Y to go up" and you actually find it causes Y to go down, that IS publishable. It is only when X has no effect on Y that you will have problems.

Re:Redefine statistical significance by BeetleB on Friday September 08, 2017 @11:06AM (#15200745)
If you mean in the context of what I was saying: Nothing wrong. But if you found some interesting correlations, you should not publish it - you should predeclare the hypothesis and what you'll look for, collect more data, and then publish if the new data matches the hypothesis.

This is assuming predeclaring ever becomes the norm.