em441

I will definitely have a look through this. Thanks!

"Upgrading" to the g-test is certainly something we could implement in the future - Pearson's Chi Square was simply a starting place. I will definitely have a look over the link you provided. Thanks!

Hi! This is Emily, the stats brains behind this. I will definitely look into this and other Bayesian methods. Thanks!

Hi! This is Emily, the stats brains behind this. My research indicated that 80% significance was commonly used for A/B Testing (and of course any significance level >80% would be even more conservative). I definitely see your point of the advantage of a continuous test and I think for the expert being able to see the exact significance level for the test would be very useful. However, our thoughts were that the average user might not necessarily have the expertise to know what a "good enough" level of significance would be for any test. Rather than having to educate every user on what significance means and how to interpret it we decided that a "yes, significant" or "no, not significant" would be more easily interpreted by all, regardless of their statistical background. If there is demand for a more continuous approach, it certainly could easily be implemented.

Hi! This is Emily, the stats brains behind this. I think this is a really interesting and important question. I know this question comes up in other contexts (clinical trials) and there has been a lot of work done on how to find "stopping times" from a statistical standpoint. I'd love to see some of these statistical methods for sequential analysis (developed for clinical trials) integrated into how we do A/B testing.