News for hackers. Stuff that matters.
Re:Getting to Statistical Significance on a Shoestring Traff... by lukego on Thursday September 14, 2023 @06:41AM (#37507212)
I believe you would apply the correction for every comparison you make regardless of the conditions. It's a conservative default to avoid accidentally p-hacking.

There might be other more specific corrections that give you power in a specific case. I don't know about that, I went Bayesian somewhere around this point myself.

Re:Getting to Statistical Significance on a Shoestring Traff... by RandomLensman on Thursday September 14, 2023 @07:06AM (#37507392)
There are a bunch of procedures under the label Family-wise Error Correction, some have issues in situations with non-independence (Bonferoni can handle any dependency structure, I think).

If there are a lot of tests/comparisons could also look at controlling for the False Discovery Rate (usually increases power at the expense of more type I errors).