Re:Conservation of Intent: why A/B tests aren’t as effective...
by RA_Fisher on Tuesday July 03, 2018 @11:36PM
(#17455123)
Bingo! "A/B testing" aka completely randomized design: https://www.itl.nist.gov/div898/handbook/pri/section3/pri331...
Mountains can be moved with randomized control trials.
What are they testing for in their metrics? Is it just click rates? Well that’s a patently losing strategy in the long term.
What if whether or not you came back over time is what was being measured? Would that break some cardinal rule that all A/B tests must be done within a short time period?
Researchers has been doing A/B testing for centuries. Till now they’ve just called it having a control variable and an independent variable. Experimental design is a big deal, and something that is taught to every graduate student researcher, but I don’t see the tech blogs giving experimental design enough focus — at least not in their click-bait headlines. Sometimes people don’t design good experiments and then blame the methods.