News for hackers. Stuff that matters.
Re:How Not To Run An A/B Test by equark on Thursday December 29, 2011 @01:53PM (#3404787)
The problem is that this advice completely ignores the motivation for experimenting: optimal decision making.

If you run a test that ends inconclusively should you really just throw up your hands? And if you run a test that's quickly conclusive, should you really avoid all the profit that could be gained from immediately exploiting this knowledge?

Re:How Not To Run An A/B Test by viraptor on Thursday December 29, 2011 @03:02PM (#3405123)
Inconclusive result is a kind of result. You can test comparable designs all you want and get an inconclusive result for a long time. This means there's no big difference and that's that.

If you get significant result at 0.0005, then it's up to you - might as well stop. There's even a table in the article saying what significance is appropriate after "correction".