News for hackers. Stuff that matters.
Re:Conservation of Intent: why A/B tests aren’t as effective... by Xcelerate on Tuesday July 03, 2018 @11:39PM (#17455130)
Yeah, classical A/B testing only works if you specify the duration of time that the experiment will run in advance and stick to it. There are ways that you can continuously monitor how an experiment is performing (Google "sequential Bayesian analysis") but the math is quite a bit more complex and nuanced.

Honestly, for most startups a simple multi-armed bandit approach is probably the way to go. Don't worry about statistical significance; just throw some "lite" reinforcement learning on top of your product's aesthetics and enjoy the incremental profit. (Caveat: do not apply MAB to major product changes.)

Re:Conservation of Intent: why A/B tests aren’t as effective... by taejo on Wednesday July 04, 2018 @05:40AM (#17456203)
> Honestly, for most startups a simple multi-armed bandit approach is probably the way to go.

Doesn't this mean maintaining all the variants forever?