News for hackers. Stuff that matters.
Re:Most Winning A/B Test Results Are Illusory [pdf] by imh on Friday January 20, 2017 @12:05AM (#13441836)
You are advocating for something called sequential testing. I'm not going to go into the details, but you don't use the same metrics with sequential testing if you want to control your final error rates. That is, you can't look at the p-value on your t-test and say that's your false positive rate for the sequential test. It's important for reasons you describe, but not exactly idiot proof.

There are no comments on this comment yet.