Re:Most Winning A/B Test Results are Illusory [pdf]
by martingoodson on Monday February 24, 2014 @05:12AM
(#7289996)
I'm afraid that's not quite right. A simple python simulation will show you that a variant with -5% (ie NEGATIVE) uplift will still give a positive results around 10% of the time if you perform early stopping of the test.
[ Parent
| on HN ]
Re:Most Winning A/B Test Results are Illusory [pdf]
by paraschopra on Monday February 24, 2014 @09:56AM
(#7291068)
No matter which method you adopt, you cannot eliminate false positives entirely. You merely decrease / control the proportion of them.
Re:Most Winning A/B Test Results are Illusory [pdf]
by martingoodson on Monday February 24, 2014 @10:55AM
(#7291438)
To remove all doubt, your interpretation of the statistics is incorrect. In particular this sentence is demonstrably false: "They are directionally correct, [...] the business will still do better implementing the variation (v/s not doing anything)."