Re:The Surprising Power of Online Experiments (2017)
by jlinowski on Sunday February 24, 2019 @11:37PM
(#19242971)
1) Agreed that the smaller the effect, the more statistical power (usually from a larger sample size) you need to detect them. But to assume that all changes have tiny effects, and therefore not detectable and a waste of time, is a flawed assumption.
Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).
More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.
2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.
Once upon a time we published over 100 a/b tests here: https://www.goodui.org/evidence/ and clearly the relative effects vary (not all single changes have always a small effect).
More so, the effects of a/b tests can be further increased by grouping multiple higher confidence ideas together into a single variation.
2) Short term gains may (or may not) lead to long term disengagement. Measuring micro (shallow) and macro (deeper) metrics would be the right way to answer this.