News for hackers. Stuff that matters.

A comment on Lines of code that will beat A/B testing every time (2012). Parent comment

Re:Lines of code that will beat A/B testing every time (2012) by sweezyjeezy on Monday January 13, 2025 @10:25PM (#42693179)
Sampling 50/50 will always give you the best chance of picking the best ultimate 'winner' in a fixed time horizon, at the cost of only sampling the winning variant 50% of the time. That's true if the reward rates are fixed or not. But some changes in reward rates will also cause MAB aggregate statistics to skew in a way that they shouldn't for a 50/50 split yeah.
[ Parent | on HN ]
Re:Lines of code that will beat A/B testing every time (2012) by zeroCalories on Monday January 13, 2025 @10:59PM (#42693439)
What do you think of using the epsilon-first approach then? We could explore for that fixed time horizon, then start choosing greedy after that. I feel like the only downside is that adding new arms becomes more complicated.
[ Parent | on HN ]
Re:Lines of code that will beat A/B testing every time (2012) by hinkley on Tuesday January 14, 2025 @01:27PM (#42701551)
What percent of companies using A/B testing do you think know what the Texas Sharpshooter is and how to identify it, let alone what epsilon is or what it means?
[ Parent | on HN ]