News for hackers. Stuff that matters.
Re:A/B test improved your website's conversion rate? Not so ... by Maxion on Friday January 07, 2022 @06:08PM (#29846226)
You take three samples and two of three get shown the same thing. What happens here is both A groups will show different results until your sample grows enough that the CI window becomes small.

This helps to show the effect of a low sample from a non-uniform distribution.

A lot of people (me included) think they know statistics, but they don't.

The blogpost in OP also tries to explain the same thing - you shouldn't do statistics without understanding what it is your doing.

Re:A/B test improved your website's conversion rate? Not so ... by bananamansion on Saturday January 08, 2022 @09:55AM (#29852202)
dont you then need to run the experiment for a long long time to get the significance of it? + the site needs enough users viewing the page to run it in any reasonable amount of time.. i would think most sites dont get enough traffic to run a/a let alone ab?