Re:Getting to Statistical Significance on a Shoestring Traff...
by lukego on Thursday September 14, 2023 @05:56AM
(#37506913)
... and one such correction is the (simple, conservative, underused) Bonferroni Correction.
[ Parent
| on HN ]
Re:Getting to Statistical Significance on a Shoestring Traff...
by AlexeyMK on Thursday September 14, 2023 @06:34AM
(#37507159)
Super helpful - looked it up, will aim to apply next time!
Curious how the bonferroni correction applies in cases where the overlap is partial - IE, experiment A ran from Day 1 to 14, and experiment B ran (on the same group) from days 8 to 21. Do you just apply the correction as if there was full overlap?
Re:Getting to Statistical Significance on a Shoestring Traff...
by lukego on Thursday September 14, 2023 @06:41AM
(#37507212)
I believe you would apply the correction for every comparison you make regardless of the conditions. It's a conservative default to avoid accidentally p-hacking.
There might be other more specific corrections that give you power in a specific case. I don't know about that, I went Bayesian somewhere around this point myself.
Re:Getting to Statistical Significance on a Shoestring Traff...
by RandomLensman on Thursday September 14, 2023 @07:06AM
(#37507392)
There are a bunch of procedures under the label Family-wise Error Correction, some have issues in situations with non-independence (Bonferoni can handle any dependency structure, I think).
If there are a lot of tests/comparisons could also look at controlling for the False Discovery Rate (usually increases power at the expense of more type I errors).