Re:Investigating how the New York Times A/B tests their head...
by tqi on Thursday March 11, 2021 @11:49AM
(#26426072)
Sequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time.
Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.
I’m sure you would agree that would be a good representation.