To use your own example. If you picked a coin from said bag which is mostly full of otherwise unbiased coins, and your hypothesis is "this coin is not biased", a sequence of ten heads would be largely "incompatible" with your hypothesis (as if the hypothesis of unbiasedness was true, you should only expect this result 1/1024 times).
It is up to you to then place this result in the context of other prior assumptions etc, and say "yes it's incompatible but still very likely". The p-value does not preclude you from performing a post-hoc analysis of the base-rate if you so wish. It just so happens that the 'assume a hypothesis' approach is a useful one in the absence of all other information. Which is not so different from a Bayesian saying 'I'll use an uninformative prior and let the data speak for itself'.
But using it as a test of posterior probability, especially when prior information exists, and saying "ha see it sucks" is not a fair criticism for it (whereas criticising its use as a test of posterior probability is).
I also disagree with the 'mindblowing' example in your blogpost linked above (though I did enjoy reading it, thank you for linking!). Your example only shows the 'superiority' of bayesian reasoning because you (presumably unintentionally) misrepresented the alternative hypothesis by the wrong 'tail' (to keep things simple - one could go further say that there are more effective alternative hypothesis formulations than a 'tailed' one here, since we are not testing simply against the alternative of 'higher bias' but against a known specific bias of 90%). One's choice of alternative hypothesis is part of one's assumptions reflecting prior knowledge. The analysis is misleadingly poor for the frequentist analysis, in that it is equivalent to the bayesian analogue of saying "Assume we have a coin that gives heads 90% of the time and I toss it 10 times; let's represent this with a prior probability Binom(10, 0.1)". What? Of course you can expect such a poor prior to give wrong results with high certainty! It makes no sense to assume that if your null is not true, then you should look for prior knowledge in the lower values. Furthermore, the exact definition of a p value allows great flexibility in the definition of the 'or worse' criterion. You are free to choose a criterion that takes into account prior information (and that forms part of your assumptions). In the 'mindblowing' example, you see that as soon as you choose the ranking measure to be P/(P+Q), and use it to calculate p-values by summing over the probabilities of the 'or worse' cases with regard to that ranking, you get very clear patterns that agree with your bayesian analysis (i.e. that 0 HEADS are more compatible with the unbiased coin hypothesis, than 1 HEADS, and similarly that the reverse applies for the biased hypothesis).
For example, if you conducted a study every week for 20 years, you'd both be extremely prolific and expect to have drawn about one wrong conclusion.
The example is a case of an absurd premise (i.e., a fair coin comes up heads 10 times in a row) leading to an absurd conclusion (that the coin is biased). Of course, this is exactly the guarantee the hypothesis test provides: under robust assumptions, you'll draw the wrong conclusion only rarely.
The OP's example would also worked well enough with even just 5 heads; you'd already pass p < 0.05, despite the actual probability of having picked the biased coin still being minuscule.
You have P(H) == 1/1024, P(H|B) == 1/1. And,
Then the probability that you picked a biased coin (P(B))? 1/10^1000. The probability you picked a fair coin (P(F))? (1 - 1/10^1000)
All in all, P(B|H)? Well Bayes rule tells us P(A|B) = P(B|A) * P(A) / P(B). Or in this case, P(B|H) = 1 * 1024 / 10^1000, or approximately 1 / 10 ^ 997.
> For example, if you conducted a study every week for 20 years, you'd both be extremely prolific and expect to have drawn about one wrong conclusion.
This is a prior! In the kind of experimentation I do, I run literally tens of thousands of experiments a day.
1 / (1/6)^5 ≃ 7776
Let's use your example. Conditional probability is P(A|B), the probability of event A, given that event B was observed. What's the probability that I am a magician, given that I shuffled the deck and when you saw it it was still in new deck order?
Now certainly, there is an astronomically small chance that this was observed due to random chance. And yet if you observed this, I'd expect that you would, with relatively high confidence, believe that I stacked the deck.
If you have 10^1000+1 coins, how did you accurately ascertain that exactly 1 coin was biased and the others were not?
And if you have been able to do that, well enough to state it as fact, why do you now need to do this statistical test?
If I wanted to identify whether a particular coin was the [most] biased out of 10^1000, flipping one coin 10 times isn't a reasonable setup for the experiment.
And, my conclusion to your hypothetical experiment would be: the person who told me that there was a 99.9...% chance this coin was fair was likely incorrect about that.
Consider a bag containing two coins, one that's fair and one that always shows heads. Pick one coin at random and flip it ten times. Suppose you see ten heads. Would you be comfortable saying that you have the biases coin? Most likely you would. Formally you'd compute the p value (in this case something like 1/1024 or 2/1024 depending on if you do a one sided or two sided test). You'd compare this with some cutoff, perhaps 0.05 and claim that you can reject the null hypothesis.
So far so good.
Now let's perform that experiment again but this time instead of having one fair coin and one biased coin in the bag, suppose we have one biases coin and 10^1000 fair coins. Suppose again that you randomly pick a coin from the bag and flip it ten times. Again you see ten heads. Do you think you have the biases coin? Of course not! What's the chance that you just happened to choose the single biased coin instead of one of the 10^1000 fair coins? Practically zero. In this case it's much more likely that the ten heads were a statistical fluke from a fair coin. However, if you compute the p value you still get 1/1024 (or 2/1024 for a two sided test).
This example should be reasonably convincing that one needs access to a prior in order to correctly interpret p values. One way of looking at it is that you need to set the threshold appropriately based on the plausibility of the alternate hypothesis. Another way to phrase it is that we actually need to compute the posterior Bayesian probability of the null hypothesis being true.
There are tons of other confusing / surprising examples where a naive p-value calculation will be misleading.
I wrote about these things here: https://mindbowling.wordpress.com/2016/07/19/p-values/