News for hackers. Stuff that matters.

A comment on 20 lines of code that beat A/B testing every time. Parent comment

Re:20 lines of code that beat A/B testing every time by emiliobumachar on Tuesday May 29, 2012 @07:54PM (#4040619)
I don't think this flaw is real.

If "whatever design was most popular at that time" has a billion trials and 90% successes, and "a new superior design" has 100 trials and 97% successes (97 out of 100), than the new design is favored by the algorithm. No need to "catch up" to the absolute number of successes.

[ Parent | on HN ]
Re:20 lines of code that beat A/B testing every time by barefoot on Tuesday May 29, 2012 @10:51PM (#4041140)
Exactly what I was thinking. What are we missing?
[ Parent | on HN ]
Re:20 lines of code that beat A/B testing every time by achllies on Wednesday May 30, 2012 @01:22AM (#4041498)
What was meant was: If there is a period of time when you get a lot of visits, lots of clicks, and abnormally high CTR. This could happen due to external factors, for example if make the front page of HN. Over time, this effect will vanish, but you will be stuck with high CTR estimates for the design that was in place when this happened for a long time.
[ Parent | on HN ]