News for hackers. Stuff that matters.

A comment on Self-Optimizing A/B Tests.

Re:Self-Optimizing A/B Tests by Normal_gaussian on Tuesday October 12, 2021 @08:49AM (#28839004)
This is the classic Multi-Armed bandit problem https://en.m.wikipedia.org/wiki/Multi-armed_bandit

I like the graphs and the explanation leads the reader deeper, but it takes the naive approach to exploration without discussing trade-offs.

Tangentially, nearly every self-optimising a/b test I have code reviewed has been significantly flawed.

[ Parent | on HN ]
Re:Self-Optimizing A/B Tests by chanind on Tuesday October 12, 2021 @12:07PM (#28841414)
Thanks for pointing this out! I update the post to note that this is a Multi-Armed Bandit problem, and linked to this comment in the updates section.
[ Parent | on HN ]