News for hackers. Stuff that matters.
Re:Conservation of Intent: why A/B tests aren’t as effective... by tehlike on Tuesday July 03, 2018 @04:57PM (#17453564)
Not entirely true. You can setup your experiment to be user keyed (as it should be) and can run it for a long time, or can run the hold back for a long time

Something as basic as mod(hash(experiment_group_id or layer id, userid), 1000) would give you a user stable experiment distribution that you can analzy long term impact on.

Re:Conservation of Intent: why A/B tests aren’t as effective... by kolpa on Tuesday July 03, 2018 @06:12PM (#17453984)
Long-term A/B testing has major problems in the modern world where users can communicate with each other (both onsite and out of band) and can use multiple accounts / devices / browsers
Re:Conservation of Intent: why A/B tests aren’t as effective... by tehlike on Tuesday July 03, 2018 @10:05PM (#17454856)
communicating user is less of a concern, cross device _is_ a real concern. In some cases it's minimized with user logins, though.