News for hackers. Stuff that matters.
Re:It’s not just p=0.048 vs. p=0.052 by tgb on Wednesday September 11, 2019 @07:57AM (#20939001)
I think p-values are actually somewhat demonized and I have grown to like them more and more over time. The standard interpretation is actually overly complicated for some reason and it can be simplified to "your p-value cutoff is an upper-bound on the rate of type I errors," over the long term. That's simple and actionable and is an immediate consequence of the definition of p-values! Frankly, I don't know why text-books don't give this as the definition of p-values, and they should reserve "probability of an event at least as extreme conditional upon the null hypothesis" as the thing-you-show-to-prove-it's-a-p-value.

The current trend of saying that "cutting p-values off at a specific value is bad" makes me worry. Now you can argue that your p=0.06 result shouldn't be rejected when really we should probably be pushing for stricter standards rather than inching towards looser ones. It also destroys the nice interpretation of p-values above. P-values were literally made to be cut off - if you want to stop doing that, you need to show me a coherent philosophy of what to do instead.

What I do think is true is the problem you have where part A of the experiment suggests X so you test X more directly in part B with a weaker but more specific test and get p=0.06 and now you can't publish. That's a dumb cutoff, clearly a p=0.06 test is likely to shift our belief towards X so it does nothing but bolster part A. Typically papers do this several times and the marginal 'failure' of one step should not sink the entire ship. This is a case where a Bayesian analysis might be more useful as it can incorporate weak evidence.

But the problem I see often is not that p-values are misused but that they were junk in the first place. For example, the widely-used DESeq2 (as well as some competitors in RNA-seq differential expression analysis) will happily spit out p-values of 10^-100 for an experiment with only four replicates in each of two conditions! There is no way you can get that level of evidence from just four replicates, even if the values are 0,0,0,0 and 1e6,1e6,1e6,1e6. The assumption of normality is reasonable near the mean but gets increasingly inaccurate in the tail, which is exactly where you end up when you do things like sort 30,000 tests by their p-values. In fact taking a p-value cutoff is probably the only reasonable thing to do here - that way you'll ignore the fact that it's absurdly small and just treat it as "small enough".

Re:It’s not just p=0.048 vs. p=0.052 by BenoitEssiambre on Wednesday September 11, 2019 @08:54AM (#20939531)
That interpretation makes the test basically useless because we know a priori that any two variable for things within each others light cone affect each other at least a little. More practically, since the test tells you nothing about size of effect, it will pick up on the tiniest of bias in your experimental procedure and always reject the null if you have enough data.

From the author of the article: "The general point reminds me of my dictum that statistical hypothesis testing works the opposite way that people think it does. The usual thinking is that if a hyp test rejects, you’ve learned something, but if the test does not reject, you can’t say anything. I’d say it’s the opposite: if the test rejects, you haven’t learned anything—after all, we know ahead of time that just about all null hypotheses of interest are false—but if the test doesn’t reject, you’ve learned the useful fact that you don’t have enough data in your analysis to distinguish from pure noise."

(https://statmodeling.stat.columbia.edu/2019/08/18/i-feel-lik...)

Re:It’s not just p=0.048 vs. p=0.052 by BenoitEssiambre on Wednesday September 11, 2019 @09:05AM (#20939661)
To add to this, the problem with t-tests in not the threshold. It is that the hypothesis you are rejecting (effect is exactly 0.0000000...) is infinitesimaly small. You've rejected basically nothing of your hypothesis space.

Your null should have a width. You should always be rejecting "effect is greater than some margin" which you should have to argue is greater than any bias you might expect in your experiment. There are always at least tiny biases.

Re:It’s not just p=0.048 vs. p=0.052 by tgb on Wednesday September 11, 2019 @05:11PM (#20944761)
Compound/interval null hypotheses basically solve this effect-size problem and probably should be used more.
Re:It’s not just p=0.048 vs. p=0.052 by BenoitEssiambre on Wednesday September 11, 2019 @05:22PM (#20944854)
Yes, it should be required.
Re:It’s not just p=0.048 vs. p=0.052 by thanatropism on Wednesday September 11, 2019 @08:32AM (#20939329)
P-values are a sub-optimal but okay-ish of quantifying a Popperian hypothesis (a designed-to-be-refutable conjecture). The mathematics is not the problem, the problem is carving science (which in my view (and Quine's and others's) is pretty much defined by the unity of science) in testable morcels.

None of the great achievements of science (Newton, Darwin, Mendeleyev, etc.) were obtained on the basis of Popperian demarcationism/conjectures-and-refutations -- they were obtained by positing a large framework and patching together the empirical case for it.

Re:It’s not just p=0.048 vs. p=0.052 by lonelappde on Wednesday September 11, 2019 @08:52AM (#20939506)
Your great achievements in science exclude all real world applications (engineering, pharmaceuticals, etc), where the critical details of a hypothesis don't fit on a t-shirt
Re:It’s not just p=0.048 vs. p=0.052 by thanatropism on Wednesday September 11, 2019 @09:51AM (#20940187)
Note that I said "science", not "engineering". Engineering is driven by usefulness and profit.

Falsificationism isn't a stupid idea; it's even useful at a personal improvement level. But pharma or materials research use it because it tends to lead to good results, not because it's the very definition of what's worthwhile knowledge.

Re:It’s not just p=0.048 vs. p=0.052 by Quekid5 on Wednesday September 11, 2019 @04:55PM (#20944597)
Turns out that once you invent a system for "acceptance" (p < 0.05 or whatever), people will game it. Who knew?
Re:It’s not just p=0.048 vs. p=0.052 by lmm on Wednesday September 11, 2019 @08:52AM (#20939512)
Indeed. People keep saying Bayesianism will solve the replication crisis but as far as I can see it will make it worse: without a hard cut-off you have many more degrees of freedom to "hack" and can push ever more marginal "results".