News for hackers. Stuff that matters.
Re:Optimizing sample sizes in A/B testing by 0x31a on Friday February 07, 2020 @07:18AM (#22265263)
Even if you assume best intentions, A/B testing often comes down to a variation of a t-test, the interpretation of which can be fuzzy too - https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf (TLDR - when it comes to statistical significance you can be either wrong or vague).
Re:Optimizing sample sizes in A/B testing by srean on Friday February 07, 2020 @07:36AM (#22265375)
Why t-test in particular ? That would be applicable as long as one has two Gaussian populations with the same variance and potentially different means.

In my experience, those preconditions are violated more frequently than they are met, unless some extra steps are taken to make the data Gaussian with equal variance. Central limit theorem helps sometimes if the true variance of the populations are well behaved (not very large).

Re:Optimizing sample sizes in A/B testing by 0x31a on Friday February 07, 2020 @09:14AM (#22266075)
You don't need equal variances - https://en.wikipedia.org/wiki/Welch%27s_t-test
Re:Optimizing sample sizes in A/B testing by mattkrause on Friday February 07, 2020 @10:44AM (#22267034)
It's also surprisingly robust to non-Gaussianity.
Re:Optimizing sample sizes in A/B testing by srean on Saturday February 08, 2020 @05:52AM (#22275136)
Not if it has a heavier tail, which is common in practice. It is disastrously brittle in those cases, mainly because sample variance is a lot of garbage in those cases.
Re:Optimizing sample sizes in A/B testing by mattkrause on Sunday February 09, 2020 @12:03AM (#22280469)
Ehh, it's more complicated than that.

Asymmetrical distributions don't fare well, in part because the mean probably isn't the thing you want to be comparing anyway. Mixture distributions or data contaminated with outliers are also trouble.

On the other hand, a bunch of simulation studies have found that t-tests work pretty well for symmetric distributions, even those with moderately heavy tails like a t(6) distribution. Moreover, the power declines a lot faster than the Type I error rate (which, for whatever reason, people seem to worry about more).

I've gotten a few paper reviews where someone has made the "but it's not normal" argument. Redoing the analysis with permutation or randomization tests (or both, for one stubborn reviewer) almost never changes the resulting p-values by more than a few percent. I don't know if this reflects the fact that they're usually quite similar or that we're just fairly judicious in how we analyze data.

Re:Optimizing sample sizes in A/B testing by srean on Sunday February 09, 2020 @10:07AM (#22282439)
I would agree that its a lot more resistant than one would think, but even an innocuous Gaussian with a heavy tailed contamination confuses it.

For cases where one needs too test "is this bigger than that ?" I test for stochastic dominance rather than equality of mean.

Re:Optimizing sample sizes in A/B testing by srean on Saturday February 08, 2020 @05:51AM (#22275129)
That would be Welch test and not t-test