And this article emphasises that huge numbers of scientists and their audiences cannot correctly interpret negatives (false or otherwise).
The negative means: "Our data does not appear to show any correlation that met the criteria we chose to determine statistical significance"
This negative is frequently misinterpreted as: “We have shown with acceptable confidence that there is no correlation". (which does not inherently follow).
Therefore, adjusting the common p value threshold to P<0.01 while not correcting this widely-held cognitive error could (potentially) even worsen the problem, because people will encounter proportionally even more of the 'negative' results that they are already well-established as poor interpreters of.
The problem with p values is the opposite: in a complex system, null hypothesis is almost never true. Everything affects everything else. It's the magnitude of the effect that is important, because it can be too small to have practical implications.
If you do enough studies eventually you'll find a result with p value below 0.05 that will overestimate the magnitude of the effect by a lot, and publish it.
For example, global weather is a hugely complex system which is far from fully understood. Let's take an incredibly simple predictive model of temperature: Still, even with the most simplistic model, say, predict the current temperature by just predicting the average temperature for the current season. That's not going to be very accurate, but if you run it for a year it will almost certainly be p<.01.
Not really. You raise the probability of an inconclusive result when you could otherwise have gotten a positive. If you interpret p > threshold as “null hypothesis is true”, then you are doing the statistics wrong.
In most cases, I think a better model would be to extract an effect size such that an effect larger than the size is ruled out by the study to some degree of confidence. Currently, I read about studies that conclude that “such-and-such had no significant effect detected by this study.” Concretely, this looks like “vaccines had no significant effect on autism risk.” This may be accurate, but it’s lousy. How about “vaccines caused no more than an 0.01% increase in autism, and a bigger study could have set an even tighter limit.”
Physicists regularly do this. For example, we know that the universe has no overall curvature to a very good approximation.
p<0.01 is good when we have a good model of reality which generally works. When we have no good model, there are no good value for p. The trouble is all the hypotheses are lies. The are false. We need more data to find good hypotheses. And we think like "there are useful data, and there are useless, we need to collect useful, while rejecting useless". But we do not know what data is useful, while we have no theory.
There is an example from physics I like. Static electricity. Researchers described in their works what causes static electricity. There was a lot of empirical data. But all that data was useless, because the most important part of it didn't get recorded. The most important part was a temporality of phenomena. Static electricity worked some time after charging and then discharged. Why? Because of all the materials are not a perfect insulators, there was a process of electrical discharge, there was voltage and current. It was a link to all other known electical phenomena. But physicists missed it because they had no theory, they didn't knew what is important and what is not. They chased what was shiny, like sparks from static electricity, not the lack of sparks after some time.
We are modern people. We are clever. We are using statistics to decide what is important and what is not. Maybe it is a key, but we need to remember that it is not a perfect key.