Investigating how the New York Times A/B tests their headlines

Re:Investigating how the New York Times A/B tests their head... by tqi on Wednesday March 10, 2021 @11:33PM (#26420277)

Agree, that is absolutely not how AB tests are run. No one runs tests sequentially on 100% of traffic - that would be like a McDonalds offering two versions of the Egg McMuffin, one from 7a-11a and the second from 11a-6p, and declaring the v1 a clear winner because it had more sales.

Assignments are also almost certainly sticky based on a browser cookie.

[ Parent | on HN ]

Re:Investigating how the New York Times A/B tests their head... by megablast on Thursday March 11, 2021 @03:02AM (#26421506)

No. It’s like if McDonald’s offered a in the first half hour, and b in the second half hour.

I’m sure you would agree that would be a good representation.

[ Parent | on HN ]

Re:Investigating how the New York Times A/B tests their head... by tqi on Thursday March 11, 2021 @11:49AM (#26426072)

Sequential testing will never be truly random, unless you can guarantee that your traffic is uniform across time.

Imagine you're running an AB test on a website, where the number of customers you get increases linearly over time. If you test two features sequentially, there is no duration you could pick that is small enough to where you would not expect the second option to win.

[ Parent | on HN ]

Re:Investigating how the New York Times A/B tests their head... by hntrader on Thursday March 11, 2021 @05:29AM (#26422447)

The traffic to a restaurant isn't uniformly distributed over an hour window.

There's a peak 30-min in the morning just before work starts for most people, for example.

[ Parent | on HN ]