Consider effect sizes - this seems to be a completely different (yes important) question. Obviously the magnitude of the impact of the drug is important - but it isn't a replacement or "something to look at instead of p-value" because the chance that the results you saw are due to random variation is still important! You can see a massive effect size but if that is totally expected within your null model, then it is probably not all that exciting!
Effect sizes are a complement to some sort of hypothesis testing, but they are not a replacement.
> Prior probability.
yes, when you can effectively encode your prior probability I would say the posterior probability of seeing what you did is at least as good as p-value.
Plus the interpretation is super straight forward: CI contains zero: Not significant. If nothing else, we should make CIs the primary default instead of p-values.
But yes, you usually do have some information and it is net better to encode when possible.
If anything I wrote appeared to be saying "just look at effect size, it's the only thing that matters" then that was an error on my part. I definitely didn't intend to say that.
But I was responding to an article saying "p<0.05 considered harmful" that never mentions effect sizes at all. I think that's enough to demonstrate that, in context, "it's bad to look at p-values and ignore effect sizes" is not in fact a straw man.
Incidentally, I am not convinced that the p-value as such is often a good way to assess how likely it is that your results are due to random chance. Suppose you see an effect size of 1 unit with p=0.05. OK, so there's a 5% chance of getting these results if the true effect size is zero. But you should also care what the chance is of getting these results if the true effect size is +0.1. (Maybe the distribution of errors is really weird and these results are very likely with a positive but much smaller effect size; then you have good evidence against the null hypothesis but very weak evidence for an effect size of the magnitude you measured.) In fact, what you really want to know is what the probability is for every possible effect size, because that gives you the likelihood ratios you can use to decide how likely you think any given effect size is after seeing the results. For sure, having the p-value is better than having nothing, but if you were going to pick one statistic to know in addition to (say) a point estimate of the effect size, it's not at all clear that the p-value is what you should choose.
It would be cool if interfaces caught up so I could just draw a basic estimate of my prior and then see the posterior graph afterwards.
p-values were created to facilitate rigorous inference with minimal computation, which was the norm during the first half of the 20th century. For those who work on a frequentist framework, inference should be done using a likelihood-based approach plus model selection, e.g. AIC. It's makes it much harder to lie.
Best practice (as I understand it) is to fix the model ahead of time, before seeing the data, if possible (as in a randomized controlled trial of a new medicine, etc.).
The only question is… would you rather be up front and explicit about your assumptions, or not?
An uninformative prior is an assumption, even if it’s the one that doesn’t bias the posterior (note that here “bias” is not a bad word).
I'm also not sure why everyone perceived my comment as anti-bayesian.
But are you sure you know what I meant when I said “uninformative prior”? Because choosing an uninformative prior does not involve choosing any parameters: there is only one uninformative prior, and it’s the constant (flat) distribution which assigns equal probability to every value. It encodes no information and does not bias the posterior or result. It is the one and only mathematically-neutral prior. You can think of it as being a bit like an “identity function”.
Ultimately, I think you are strawmanning frequentism here. Just because the log likelihood is sometimes the same as the map does not imply that they have the same meaning. This is why computed uncertainties of both approaches are often not the same and have a not-so-subtle difference in their interpretation. The one computes uncertainty in belief, the other imprecision of an experiment. You can't summarize that with "do you want to be explicit about assumptions".
I am not straw-manning anything: bayesian methods are a generalization of frequentist methods. The equality / isomorphism to frequentist methods, in the special case of the uninformative prior, is commonly demonstrated in introductory (undergrad-level) bayesian textbooks and is in fact trivial. One need not even talk of any infinities: if you’re doing discrete observations, infinities never show up (and in that case the prior normalizes just fine). And if you’re curve-fitting (ie.: using parametric methods), the “infinite” line goes away as soon as you multiply your uninformative prior with your likelihood.
Ive worked with clinical trials and nobody says “oh, p < 0.05 so we should use this drug”.
Doctors do a much more sophisticated deep dive into the data and statistical significance is maybe 1 of a 10 factors they consider.
And hell, even if it is statistically significant, doctors may not use a drug for many other reasons.
It seems like the people who criticize p-values aren't that familiar with how they are actually used.
It's immensely frustrating that we haven't made a lot of progress since Cohen's (1962) paper.
1. Typically we use p-values to construct confidence intervals, answering the concern about quantifying the effect size. (That is, the confidence interval is the collection of all values not rejected by the hypothesis test.)
2. P-values control type I error. Well-powered designs control type I and type II error. Good control of these errors is a kind of minimal requirement for a statistical procedure. Your example shows that we should perhaps consider more than just these aspects, but we should certainly be suspicious of any procedure that doesn't have good type I and II error control.
3. This is a problem with any kind of statistical modeling, and is not specific to p-values. All statistical techniques make assumptions that generally render them invalid when violated.
But looking at the practical application, in particular the replication crisis, specification curve analysis, de facto power of published studies and many more, we see that there is an immense practical problem and p-values are not making it better.
We need to criticize p-values and NHST hard, not because they cannot be used correctly, but because they are not used correctly (and are arguably hard to use right, see the Gigerenzer paper I linked).
(1) p-values make significance the target, and thus create incentives for underpowered studies, misspecified analyses, early stopping (monitoring significance while collecting data), and p-hacking.
(2) p-values separate crucial pieces of information. It represents a highly specific probability (of the observed data, given the null hypothesis is true), but does not include effect size or a comprehensive estimate of uncertainty. Thus, to be useful, p-values need to be combined with effect sizes and ideally simulations, specification curves, or meta-analyses.
Thus my primary problem with p-values is that they are an incomplete solution that is too easy to use incorrectly. Ultimately, they just don't convey enough information in their single summary. CIs, for example, are just as simple to communicate, but much more informative.
So I think you are correct that properly powering studies is the crucial thing, but the incentives are against fixing this as long as lone p-values are publishable.
That’s why rust came along, to stop that behaviour, you simply can’t make that mistake, and hence the point is maybe there’s a better test to use than p value as a standard.
An area I follow closely is exercise science and I am amazed that researchers are able to get grants for some of the research they do. For instance, sometimes researchers aim to compare, say, the amount of hypertrophy on one training program versus another. They’ll use a study or maybe 15 individuals. The group on intervention A will experience 50% higher hypertrophy than intervention B, at p=0.08, and they’ll conclude that there’s no difference in hypertrophy between the two protocols rather than suggesting an increase in statistical power.
Another great example is studies whose interventions fail to produce any muscle growth in new trainees. They’ll compare two programs, one with, say, higher training volume, and one with lower training volume. Both fail to produce results for some reason. They conclude that variable is not important, rather than perhaps concluding that their interventions are poorly designed since a population that is begging to gain muscle couldn’t gain anything from it.
No. It's more about why choosing a value to serve as the default choice is a bad idea in the first place. The specific value chosen as the default itself (i.e., 0.05 in this case) is irrelevant.
The idea is that the value you choose should reflect some prior knowledge about the problem. Therefore choosing 5% all the time would be somewhat analogous to a bayesian choosing a specific default gaussian as their prior: it defeats the point of choosing a prior, and it's actively harmful if it sabotages what you should actually be using as a prior, because it's not even uninformative, it's a highly opinionated prior instead.
As for points 1 to 3, technically I agree, but there's a lot of misdirection involved.
The point on effect sizes is true (and indeed something many people get wrong), but it is contrived to make effect sizes more useful than p-values. In which case, the obvious answer is, you should be choosing an example where reporting a p-value is more important than an effect size. One way to look at a p-value is as a ranking, which would be useful for comparing between effect sizes of incomparable units. Is a student with a 19/20 grade from a european school better than an american student with a 4.6 GPA? Reducing the compatibility scores to rankings, can help you compare these two effect sizes immediately.
Prior probability, similarly. "If" you interpret things as you did, then yes, p-values suck. But you're not supposed to. What the p-value tells you is "the data and model are 'this much' compatible". It's up to you then to say "and therefore" vs "must have been an atypical sample". In other words, there is still space for a prior here. And in theory, you are free to repurpose the compatibility score of a p-value to introduce this prior directly (though nobody does this in practice).
Regarding p=0.05 vs p=0.001 not mattering; of course they do. But only if they're used as compatibility rankings as opposed to decision thresholds. If you compare two models, and one has p=0.05 and the other has p=0.001, this tells you two things: a) they are both very incompatible with the data, b) the latter is a lot more incompatible than the former. The problem is not that people use p-values, the problem that people abuse them, to make decisions that are not necessarily crisply supported by the p-values used to push them. But this could be said of any metric. I have actively seen people propose "deciding in favour of a model if the Bayes Factor is > 3". This is exactly the same faulty logic, and the fact that BF is somehow "bayesian" won't protect subsequent researchers who use this heuristic from entering a new reproducibility crisis.
I agree, in case it needs to be said, that if you are doing hypothesis testing then you should adapt your choice of thresholds to your (explicit or implicit) prior. That was approximately the second of the three points I made.
"You should be choosing an example where reporting a p-value is more important than an effect size." Why should I be doing that? I think that almost always both are important (more precisely, I think that almost always thinking of what you've learned from an experiment into a p-value and a point-estimate effect size is suboptimal, but you want the information both of those things are trying to tell you). And if I've correctly understood what you say about rankings, I completely disagree with it; almost none of the time is the p-value what you should be ranking things on. Again, consider those two medications in my hypothetical example: I do not believe any reasonable person would think that the right way to rank them relative to one another matches the p-values. (For comparing two individual students? Yeah, maybe, kinda. But approximately 0% of cases where people compute p-values are analogous to that.)
I agree that picking any single metric and applying brainless black-and-white rules using it is liable to get you in trouble, whatever that metric is. But some metrics are better than others, and a fixed p-value threshold (1) has been a widespread common practice and (2) is for most purposes a really bad idea.
> I completely disagree with it; almost none of the time is the p-value what you should be ranking things on.
Not necessarily true; people compare models on the basis of their p-values all the time; but in any case, that's not what I was referring to here. I was saying the p-value in itself expresses a ranking. That's just what a p-value is by definition: a ranking of the particular compatibility score of the data-event under consideration, with respect to the whole domain of compatibility scores resulting from all possible data-events considered.
And yes, when the score itself is more important than the ranking, it's prudent to focus on that; when the p-value "ranking" is more important, it's prudent to focus on that instead; ideally, people should consider both.
Having said that, the medical example is a weird one. There's a specific sentence there that I have a problem with:
> It's more uncertain because the sample size is smaller [...] I would definitely prefer the second drug.
Perhaps this is simply a misnomer, since one can only be so precise when using language, but as expressed above, I feel this is wrong, as it alludes to an expression of posterior probability, which is not information given to you by a p-value in this setting. More specifically you should be saying it's not "precise". The subtle difference being, you are interpreting this as a "despite the wide range, it's most likely to be around 0.1, so I'll take my chances", whereas what the confidence interval is really telling you is "the data would be atypical for models outside this large range, but not inside", meaning there is an almost equal chance of the drug harming you as there is giving you a benefit. So, no, I wouldn't prefer the 2nd drug; I'd actually prefer the 1st one which has a guaranteed, albeit smaller benefit. This is a very different statement from a credible interval which more or less would state that 0.1 is the most likely value.
So in fact, the p-value would be a better measure of comparison here as well. You're comparing the compatibility of two different data-events (which in this case reflects samplings) against the same (null) hypothesis; so you should prefer the event that is least compatible with the null (and ideally, you should propose an alternative hypothesis, and confirm that it demonstrates higher compatibility with that one as well). But having said this, comparing p-values typically comes in the reverse scenario, were one has a single data-event, and wants to compare two different hypotheses against the same data-event, since that gives information on which model is most compatible with what you observed.
Anyway. Sorry, I know you know, I'm just enjoying the exposition :)
My favourite example is to do that with normal distributions and the posterior ends up as "let's meet in the middle and agree that we both were wrong" -a very low variance posterior with mean right in the middle between prior and data.
1. Effect sizes.
Suppose you are a doctor or a patient and you are interested in two drugs. Both are known to be safe (maybe they've been used for decades for some problem other than the one you're now facing). As for efficacy against the problem you have, one has been tried on 10000 people, and it gave an average benefit of 0.02 units with a standard deviation of 1 unit, on some 5-point scale. So the standard deviation of the average over 10k people is about 0.01 units, the average benefit is about 2 sigma, and p is about 0.05. Very nice.
The other drug has only been tested on 100 people. It gave an average benefit of 0.1 unit with a standard deviation of 0.5 units. Standard deviation of average is about 0.05, average benefit is about 2 sigma, p is again about 0.05.
Are these two interventions equally promising? Heck no. The first one almost certainly does very little on average, and does substantially more harm than good about half the time. The second one is probably about 5x better on average, and seems to be less likely to harm you. It's more uncertain because the sample size is smaller, and for sure we should do a study with more patients to nail it down better, but I would definitely prefer the second drug.
(With those very large standard deviations, if the second drug didn't help me I would want to try the first one, in case I'm one of the lucky people it gives > 1 unit of benefit to. But it might well be > 1 unit of harm instead.)
Looking only at p-values means only caring about effect size in so far as it affects how confident you are that there's any effect at all. (Or, e.g., any improvement on the previous best.) But usually you do, in fact, care about the effect size too.
Here's another way to think about this. When computing a p-value, you are asking "if the null hypothesis is true, how likely are results like the ones we actually got?". That's a reasonable question. But you will notice that it makes no reference at all to any not-null hypothesis. p < 0.05 means something a bit like "the null hypothesis is probably wrong" (though that is not in fact quite what it means) but usually you also care how wrong it is, and the p-value won't tell you that.
2. Prior probability.
The parenthetical remark in the last paragraph indicates another way in which the p-value is fundamentally the Wrong Thing. Suppose your null hypothesis is "people cannot psychically foretell the future by looking at tea leaves". If you test this and get a p=0.05 "positive" result, then indeed you should probably think it a little more likely than you previously did that this sort of clairvoyance is possible. But if you are a reasonable person, your previous opinion was a much-much-less-than-5% chance that tasseomancy actually works[1], and when someone gets a 1-in-20 positive result you should be thinking "oh, they got lucky", not "oh, it seems tasseomancy works after all".
[1] By psychic powers, anyway. Some people might be good at predicting the future and just pretend to be doing it by reading tea leaves, or imagine that that's how they're doing it.
3. Model errors.
And, of course, if someone purporting to read the future in their tea leaves does really well -- maybe they get p=0.0000001 -- this still doesn't oblige a reasonable person to start believing in tasseomancy. That p-value comes from assuming a particular model of what's going on and, again, the test makes no reference to any specific alternative hypothesis. If you see p=0.0000001 then you can be pretty confident that the null hypothesis's model is wrong, but it could be wrong in lots of ways. For instance, maybe the test subject cheated; maybe that probability comes from assuming a normal distribution but the actual distribution is much heavier-tailed; maybe you're measuring something and your measurement process is biased, and your model assumes all the errors are independent; maybe there's a way for the test subject to get good results that doesn't require either cheating or psychic powers.
None of these things is helped much by replacing p=0.05 with p=0.001 or p=0.25. They're fundamental problems with the whole idea that p-values are what we should care about in the first place.
(I am not claiming that p-values are worthless. It is sometimes useful to know that your test got results that are unlikely-to-such-and-such-a-degree to be the result of such-and-such a particular sort of random chance. Just so long as you are capable of distinguishing that from "X is effective" or "Y is good" or "Z is real", which it seems many people are not.)