Re:A/B testing gets misused to juice metrics in the short term
by Kaotique on Tuesday July 12, 2022 @11:03AM
(#32071019)
There are many other metrics that do not involve AB testing. You can just survey customer experience before, during and after a purchase for example. I never said to throw out all metrics.
With AB testing your are optimising for a specific outcome. Usually higher conversion. As pointed out in the article eventually you'll end up with a bunch of colourful buttons and scary texts that persuade the user to click. A lot of the "only 2 seats/rooms available" are lies to scare the user into a conversion.
Re:A/B testing gets misused to juice metrics in the short term
by josefx on Tuesday July 12, 2022 @10:47AM
(#32070796)
But does AB testing provide the only or even best metric for that? It probably is the cheapest way requiring the least engagement with the lab mice.
Re:A/B testing gets misused to juice metrics in the short term
by treis on Tuesday July 12, 2022 @06:04PM
(#32076276)
AB testing is how you isolate a change and measure the impact. It's the only real way to be able to associate cause and effect. Best you can do otherwise is measure something over time while making changes. You can try and correlate changes with outcomes but it's hard to be sure the change is what drove the outcome.
Re:A/B testing gets misused to juice metrics in the short term
by oba2311 on Wednesday July 13, 2022 @07:10AM
(#32081098)
That sounds pretty accurate. Anyone who's gone through some econometric classes would know that split testing is (almost) the only way to get to (close to) causality - all the rest is black magic called correlation which can also lead you to the conclusion that Nicolas Cage's movies lead to people drowning:
https://www.thenationalnews.com/uae/nicolas-cage-movies-link....
https://www.wnycstudios.org/podcasts/otm/articles/spurious-c...
Without a metric to say what is "better" and a method to measure it this is empty advice.