Re:Five ways to reduce variance in A/B testing
by wodenokoto on Sunday September 29, 2024 @02:40PM
(#41689415)
> 2. Making sure your experiment runs in 7-day increments. Averaging out weekly seasonality can be important in reducing variance but also ensures your results accurately predict the effect of a full rollout.
There are of course many seasonalities: day/nigh, weekly, monthly, yearly seasonality, so it can be difficult to decide how broad you want to collect data. But I remember interviewing at a very large online retailer and they did their a/b tests in an hour because they "would collect enough data points to be statistical significant" and that never sat right with me.
There are of course many seasonalities: day/nigh, weekly, monthly, yearly seasonality, so it can be difficult to decide how broad you want to collect data. But I remember interviewing at a very large online retailer and they did their a/b tests in an hour because they "would collect enough data points to be statistical significant" and that never sat right with me.