News for hackers. Stuff that matters.
Re:How Optimizely Almost Got Me Fired (2014) by cle on Saturday January 09, 2016 @05:38PM (#10873174)
One underused but valuable use of A/B testing is to safely roll out new features--even if you aren't really interested in optimization. I've seen many cases where A/B testing has caught bugs that would have cost the company a significant amount of money and customer trust.

The overall utility of A/B tests is a complex topic and largely depends on the skill of the experimenter. Well-designed experiments are extraordinarily useful. A/B tests are one tool for product development, but they can't replace it.

Re:How Optimizely Almost Got Me Fired (2014) by jacalata on Saturday January 09, 2016 @06:24PM (#10873294)
I wouldn't call that A/B testing. It looks pretty similar under the hood but you'd normally call that a phased or gradual rollout and have different goals and metrics for it.
Re:How Optimizely Almost Got Me Fired (2014) by ampersandy on Saturday January 09, 2016 @10:57PM (#10874075)
As you said, what cle described is a gradual rollout. You can do both at the same time though -- simply start your test on X% of the population and then roll it out to more people as you run your test. This is also a great way to run long-term holdouts.