News for hackers. Stuff that matters.
Re:How Not To Run An A/B Test by timr on Thursday December 29, 2011 @03:37PM (#3405277)
The Dunn-Sidak correction is preferred over the Bonferroni correction -- it's much less conservative, and will allow the finding of significance in situations that Bonferroni would miss.

Another thing: depending on how your problem is structured, it might be a bit confusing to think of these as corrections for the number of hypotheses. I like to think of them in terms of the number of planned/unplanned comparisons that are being performed: you can do an experiment with a single stated hypothesis, yet still need to use these corrections if you perform "unplanned" comparisons using the data (a.k.a "data mining", or "a data fishing expedition") later on.

This is a really great summary of planned vs. unplanned comparisons and why they matter:

http://udel.edu/~mcdonald/statanovaplanned.html

There are no comments on this comment yet.