I agree, in case it needs to be said, that if you are doing hypothesis testing then you should adapt your choice of thresholds to your (explicit or implicit) prior. That was approximately the second of the three points I made.
"You should be choosing an example where reporting a p-value is more important than an effect size." Why should I be doing that? I think that almost always both are important (more precisely, I think that almost always thinking of what you've learned from an experiment into a p-value and a point-estimate effect size is suboptimal, but you want the information both of those things are trying to tell you). And if I've correctly understood what you say about rankings, I completely disagree with it; almost none of the time is the p-value what you should be ranking things on. Again, consider those two medications in my hypothetical example: I do not believe any reasonable person would think that the right way to rank them relative to one another matches the p-values. (For comparing two individual students? Yeah, maybe, kinda. But approximately 0% of cases where people compute p-values are analogous to that.)
I agree that picking any single metric and applying brainless black-and-white rules using it is liable to get you in trouble, whatever that metric is. But some metrics are better than others, and a fixed p-value threshold (1) has been a widespread common practice and (2) is for most purposes a really bad idea.
> I completely disagree with it; almost none of the time is the p-value what you should be ranking things on.
Not necessarily true; people compare models on the basis of their p-values all the time; but in any case, that's not what I was referring to here. I was saying the p-value in itself expresses a ranking. That's just what a p-value is by definition: a ranking of the particular compatibility score of the data-event under consideration, with respect to the whole domain of compatibility scores resulting from all possible data-events considered.
And yes, when the score itself is more important than the ranking, it's prudent to focus on that; when the p-value "ranking" is more important, it's prudent to focus on that instead; ideally, people should consider both.
Having said that, the medical example is a weird one. There's a specific sentence there that I have a problem with:
> It's more uncertain because the sample size is smaller [...] I would definitely prefer the second drug.
Perhaps this is simply a misnomer, since one can only be so precise when using language, but as expressed above, I feel this is wrong, as it alludes to an expression of posterior probability, which is not information given to you by a p-value in this setting. More specifically you should be saying it's not "precise". The subtle difference being, you are interpreting this as a "despite the wide range, it's most likely to be around 0.1, so I'll take my chances", whereas what the confidence interval is really telling you is "the data would be atypical for models outside this large range, but not inside", meaning there is an almost equal chance of the drug harming you as there is giving you a benefit. So, no, I wouldn't prefer the 2nd drug; I'd actually prefer the 1st one which has a guaranteed, albeit smaller benefit. This is a very different statement from a credible interval which more or less would state that 0.1 is the most likely value.
So in fact, the p-value would be a better measure of comparison here as well. You're comparing the compatibility of two different data-events (which in this case reflects samplings) against the same (null) hypothesis; so you should prefer the event that is least compatible with the null (and ideally, you should propose an alternative hypothesis, and confirm that it demonstrates higher compatibility with that one as well). But having said this, comparing p-values typically comes in the reverse scenario, were one has a single data-event, and wants to compare two different hypotheses against the same data-event, since that gives information on which model is most compatible with what you observed.
Anyway. Sorry, I know you know, I'm just enjoying the exposition :)
No. It's more about why choosing a value to serve as the default choice is a bad idea in the first place. The specific value chosen as the default itself (i.e., 0.05 in this case) is irrelevant.
The idea is that the value you choose should reflect some prior knowledge about the problem. Therefore choosing 5% all the time would be somewhat analogous to a bayesian choosing a specific default gaussian as their prior: it defeats the point of choosing a prior, and it's actively harmful if it sabotages what you should actually be using as a prior, because it's not even uninformative, it's a highly opinionated prior instead.
As for points 1 to 3, technically I agree, but there's a lot of misdirection involved.
The point on effect sizes is true (and indeed something many people get wrong), but it is contrived to make effect sizes more useful than p-values. In which case, the obvious answer is, you should be choosing an example where reporting a p-value is more important than an effect size. One way to look at a p-value is as a ranking, which would be useful for comparing between effect sizes of incomparable units. Is a student with a 19/20 grade from a european school better than an american student with a 4.6 GPA? Reducing the compatibility scores to rankings, can help you compare these two effect sizes immediately.
Prior probability, similarly. "If" you interpret things as you did, then yes, p-values suck. But you're not supposed to. What the p-value tells you is "the data and model are 'this much' compatible". It's up to you then to say "and therefore" vs "must have been an atypical sample". In other words, there is still space for a prior here. And in theory, you are free to repurpose the compatibility score of a p-value to introduce this prior directly (though nobody does this in practice).
Regarding p=0.05 vs p=0.001 not mattering; of course they do. But only if they're used as compatibility rankings as opposed to decision thresholds. If you compare two models, and one has p=0.05 and the other has p=0.001, this tells you two things: a) they are both very incompatible with the data, b) the latter is a lot more incompatible than the former. The problem is not that people use p-values, the problem that people abuse them, to make decisions that are not necessarily crisply supported by the p-values used to push them. But this could be said of any metric. I have actively seen people propose "deciding in favour of a model if the Bayes Factor is > 3". This is exactly the same faulty logic, and the fact that BF is somehow "bayesian" won't protect subsequent researchers who use this heuristic from entering a new reproducibility crisis.