To name just a few of the NHST/p-value flaws:
1- I'm interested in P(H1 | data) but I get P(data | Ho). Contrary to popular belief P(data|Ho) != P(Ho|data). Let alone that conclusions about P(H1|data) can be drawn.
2- it is vulnerable to wrong interpretations. * No, a 95% confidence interval (a,b) does NOT mean there is a 95% chance that Mu is in (a,b). * No, p=0.04 does not mean they is a 96% chance that H1 is true.
3- the p-value depends on the intentions of the scientist. If you end your experiment after 80 observations, as planned, your p-value is different from that of an experiment that ended unplanned after 80 observations. So the same data have different evidential power, influenced by results you did not see in experiments you did not do. This is very unsatisfactory.
4- the idea of "an effect that exists or does not exist", based on some arbitrary threshold. The reality is, in many cases, uncertainty and variation. In group A I see effects of medicine A, with lots of variation between persons. In group B I see varying effects of medicine B. Then I introduce uncertainty by drawing random samples from A and B. Let's day I used those samples to make an inference: is, on average, medicine A better than medicine B ? Matras like "there is an effect, or there isn't" are not very helpful. Statistics should be about quantifying uncertainty rather than give false yes/no statements.
5. Basing decisions and knowledge on the data only makes t vulnerable to outliers, unlucky samples and so on. And why should you NOT use information, when it's there ?
The American Statistical Association actually put out a statement this year on the issue of p-value. You can find the whole article here http://dx.doi.org/10.1080/00031305.2016.1154108 but here are the main points:
1) P-values can indicate how incompatible the data are with a specified statistical model.
2) P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
3) Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
4) Proper inference requires full reporting and transparency.
5) A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6) By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
As an additional curiosity, the group of writers was not completely unconflicted coming up those definitions and the article contains a number of supplemental articles by the individual authors to clarify/dispute some of the points made.
[edit] fixed formatting
Perhaps this is nice fur Quality Asurance in factories. Where I do repeated measurements and where I want a simple YES or NO.
But science usually asks: "what can I learn from this specific data? I don't do 100 samples and I'm not interested in being "not to far off moist of the time". I want a best estimation based on this specific sample".
NHST does not give that answer. Bayes does.
The problem of induction is real and unavoidable in the general case, but there is no such thing as "the probability that there is a real effect". Either there is a "real effect" or there is not.
It might be possible to find a "probability" that you observed a "real effect".
>The problem of induction was solved, in principle, by the Reverend Thomas Bayes in the middle of the 18th century.
The problem of induction is fundamentally unsolvable (hence, "problem"). The article just states that it was solved and never mentions it again. Is it a widely held view that induction was solved by Bayes? Does anyone know where I can read more detailed claims about how people believe Bayes solved induction?