p-Hacking and False Discovery in A/B Testing

Re:p-Hacking and False Discovery in A/B Testing by Jabbles on Sunday July 22, 2018 @02:20PM (#17588037)

Presumably using your method takes longer/requires more samples than a method that only checks once?

[ Parent | on HN ]

Re:p-Hacking and False Discovery in A/B Testing by srean on Monday July 23, 2018 @05:42AM (#17591594)

I haven't looked at the KDD paper, but in general it is the other way round. With sequential hypothesis testing expect to need less data on average.

[ Parent | on HN ]

Re:p-Hacking and False Discovery in A/B Testing by computerphage on Wednesday July 25, 2018 @01:18AM (#17607052)

That's highly counter-intuitive to me. Can you try to motivate why that's the case?

My intuition is that you could use any sequential (which I translated to online) technique could be used in a non-sequential context. By that reasoning, there's no way a sequential technique could do better, at best it could be the same.

[ Parent | on HN ]

Re:p-Hacking and False Discovery in A/B Testing by srean on Thursday July 26, 2018 @02:53PM (#17619897)

This is 1940s stuff. Checkout Wald.

Short answer: in sequential testing you can ask at intermediate stages whether a satisfactory confidence has been reached. If yes you are done and if not you can continue. On average you will hit a 'yes' sooner. For non sequential you cannot do this if you care about correctness (). So the sample size needs to be pessimistic for non-sequential protocols and then you are bound to that commitment.

() If your method ensures correctness even after inspection at intermediate stages then its a sequential method by definition. There is some confusion in literature about Bayesian and sequential. They are orthogonal concepts. Both Bayesian and Frequentist test of hypothesis can be sequential

[ Parent | on HN ]

Re:p-Hacking and False Discovery in A/B Testing by computerphage on Tuesday July 31, 2018 @03:29AM (#17651226)

Ah! I get it. Thank you!

[ Parent | on HN ]