News for hackers. Stuff that matters.
Re:Why small tweaks and split testing don't work, and what t... by tobr on Thursday November 29, 2018 @02:13PM (#18562947)
How many impressions were you running if you were able to measure a 2% difference?
Re:Why small tweaks and split testing don't work, and what t... by chudi on Thursday November 29, 2018 @02:28PM (#18563115)
If you want to be completely sure, a really small amount, lets say your baseline is 20% to detect a 2% difference you just need 2 x 6,347 views for each test

Check this tools[0] and this article [1]

[0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;8...

[1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html

Re:Why small tweaks and split testing don't work, and what t... by jermaustin1 on Thursday November 29, 2018 @02:34PM (#18563180)
When we implemented split-testing-by-default baked into the software itself, we saw maybe 1000 visitors a week from various sources. Our free conversion rate was at about 3% and once we were "fully optimized" we were closer to 10-15%, and our paid conversions were about 10-30% of the free users depending on the month (December and January always sucked).

Most of these numbers were measured after at least a week of leaving the experiments running.

We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.