News for hackers. Stuff that matters.
Re:A Better Approach to A/B Test Analysis by tedsanders on Wednesday May 28, 2014 @06:28PM (#7813371)
I totally agree. And I think Bayesian methods (which compute the likelihood of the model given the data) tend to work much better for these rolling data applications than frequentist methods (which compute the likelihood of the data given the model). The problem for frequentist methods here is that when data is constantly rolling in it's hard to specify the space of all possible data collected (because traditional bounds like the number of data points no longer work).

Here's a link to a paper on a Bayesian approach to the multi-armed bandit problem: http://onlinelibrary.wiley.com/doi/10.1002/asmb.874/abstract...

Re:A Better Approach to A/B Test Analysis by em441 on Wednesday May 28, 2014 @06:45PM (#7813446)
Hi! This is Emily, the stats brains behind this. I will definitely look into this and other Bayesian methods. Thanks!
Re:A Better Approach to A/B Test Analysis by btilly on Wednesday May 28, 2014 @08:03PM (#7813814)
I would also suggest switching from chi-square to the g-test.

If you look at the history, I believe that you'll find that Pearson originally came up with the g-test as an approximation to an exact test, and then found the chi-square as an easier to compute alternative. This mattered back in the days of pencil and paper, but there is no excuse today to use the worse technique.

I'm going to avoid long discussions about the advisability of taking multiple looks at results with a classical statistical test. But see my incomplete series at http://elem.com/~btilly/ab-testing-multiple-looks/index.html for some of the considerations.

I never got into Bayesian statistics in there. In general they depend on the existence of a prior distribution. Careful treatments will talk about this. Sloppy ones assume one, don't talk about the one that they assume, and then quote results without letting you know about this important assumption. As long as you accept that assumption, they work well. But sometimes can be confusing to explain. (Until people "get" it. Then it can become irritating getting them to STOP explaining it!)

If you want to discuss these issues more, my email is my name at gmail.com.

Re:A Better Approach to A/B Test Analysis by em441 on Wednesday May 28, 2014 @08:52PM (#7813978)
"Upgrading" to the g-test is certainly something we could implement in the future - Pearson's Chi Square was simply a starting place. I will definitely have a look over the link you provided. Thanks!