Re:A/B Testing, from scratch
by btilly on Thursday December 03, 2015 @06:05PM
(#10673398)
My scratchbook. :-)
I keep meaning to write it up.
Mathematically it is similar http://www.evanmiller.org/sequential-ab-testing.html but with the difference that Evan produced a stopping rule that is a 95% confidence interval, while mine is stopping at 95% confidence that you won't come to a different conclusion by the end of the test. Otherwise the idea of the analysis is the same.
(Note, under the null hypothesis reaching sqrt(N) then crossing 0 is exactly as hard as reading 2 sqrt(N) as can be seen by taking any possible sequence that does one, and swapping the values after first reaching sqrt(N) to get a sequence that does the other.)
I keep meaning to write it up.
Mathematically it is similar http://www.evanmiller.org/sequential-ab-testing.html but with the difference that Evan produced a stopping rule that is a 95% confidence interval, while mine is stopping at 95% confidence that you won't come to a different conclusion by the end of the test. Otherwise the idea of the analysis is the same.
(Note, under the null hypothesis reaching sqrt(N) then crossing 0 is exactly as hard as reading 2 sqrt(N) as can be seen by taking any possible sequence that does one, and swapping the values after first reaching sqrt(N) to get a sequence that does the other.)