News for hackers. Stuff that matters.
Re:How Not To Run An A/B Test by laughinghan on Thursday December 29, 2011 @12:56PM (#3404506)
Is it just me, or does that table look like

    req.sig = (intended significance)/(#hypoth)
? Which is kind of a simpler formula.

Edit: Following your link, the table you listed is indeed of the Bonferroni correction, and the formula is as I stated. The formula you stated is actually of the Sidak correction, which "is often confused with the Bonferroni correction", according to your link.

Re:How Not To Run An A/B Test by timr on Thursday December 29, 2011 @03:37PM (#3405277)
The Dunn-Sidak correction is preferred over the Bonferroni correction -- it's much less conservative, and will allow the finding of significance in situations that Bonferroni would miss.

Another thing: depending on how your problem is structured, it might be a bit confusing to think of these as corrections for the number of hypotheses. I like to think of them in terms of the number of planned/unplanned comparisons that are being performed: you can do an experiment with a single stated hypothesis, yet still need to use these corrections if you perform "unplanned" comparisons using the data (a.k.a "data mining", or "a data fishing expedition") later on.

This is a really great summary of planned vs. unplanned comparisons and why they matter:

http://udel.edu/~mcdonald/statanovaplanned.html