News for hackers. Stuff that matters.
Re:p-Hacking and False Discovery in A/B Testing by cepth on Sunday July 22, 2018 @10:49AM (#17587021)
If you chose a p-value threshold of 0.8, and your tested result came in around there, that would suggest that your null hypothesis had a ~80% chance of being true. So in any case, you do not have a strong conclusion.

I agree that over reliance on a single metric, like a p-value, gets us Goodhart’s Law type problems.

In econometrics, and really any other statistics adjacent field, if you’ve correctly estimated your standard errors, and are using something like https://en.m.wikipedia.org/wiki/Newey%E2%80%93West_estimator where appropriate, there is nothing wrong with using a p-value as a general approximation of significance.

There are no comments on this comment yet.