If you end the test the moment the data reaches 95% significance, it will show a difference about 50% of the time for the same email. Many people make this mistake.
A 95% confidence interval doesn't mean much if you dont follow good statistical practices.
A small sample size would lower your power to detect meaningful differences, which the original scenario doesn't have (by definition).
(If distributional assumptions, etc, are violated, then that's a different story!)
https://heapanalytics.com/blog/data-stories/dont-stop-your-a...
The point I was trying to make was you can decide to run a test with a very small sample (e.g. n = 5), and it will still have the level of type 1 power you set if you chose a significance level of .05.
> You need to make sure you have enough samples in order to know if you rejected the null hypothesis by chance.
You do this when you decide the significance level (e.g. .05). The value needed to reject, given a significance level, is a function of sample size.
The definition of Type 1 error on wikipedia has a good explanation of this:
What you're saying is perhaps correct but it's not how AB testing is done in the real world. I think many tests show a difference for random reasons.
b) That's assuming you are using statistics right. It's quite hard to use statistics right.
Did you know, in experimentation programs run by Microsoft, Google and Amazon, roughly two thirds of their ideas have no impact or hurt the metrics they were designed to improve? And yet rookie web Devs or marketing assistants "know" better.
Source: https://www.google.com.au/url?sa=t&source=web&rct=j&url=http... (Read section three of this paper by a Microsoft distinguished engineer)
I'll point out a major difference, though: Microsoft, Google, and Amazon are already highly optimized. They've had millions of man-hours put into optimizing everything from product design to UI to UI element positioning to wording to colors to fonts. When you get a new, beneficial result in a change to Google, it's usually because the environment has changed while you weren't looking, and the user is different.
That doesn't apply to a new business that's just learning how to sell their product. In a new business, by definition, you've spent zero time micro-optimizing your message & product. You can get really big wins almost by accident, if you happen to have stumbled into a product category where users actually want what you're building.
Obviously some segments are going to open/click/spam at higher rates, but that's fine and controllable with a customer rating. So when you split your users, ensure an even makeup of customers, and your tests will be fine.
Because the ones who weren't going to convert never came back
Check this tools[0] and this article [1]
[0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;8...
[1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html
Most of these numbers were measured after at least a week of leaving the experiments running.
We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.
I am no longer with the company, but around 10 years ago, I turned a failing consultancy into an SEO SaaS company. Turning revenues from -40k/month to 90+k/month after laying off more than half the staff when we lost our last client and deciding to take our destiny into our own hands building a suite of SEO products.
We ran split tests on almost every design change. They weren't always tiny, some were entire page layouts. But even on things like changing the header text saw measurable differences.
I built the ability to split test into every facet of our framework. We had the ability to split test pages, transactional emails, marketing emails, and we would tweak everything all the time. If we noticed group a wasn't performing as well as b, we would drop a.
There were some tiny things that made major differences:
- Email "From" being a person's name and not just the service bumpped onboarding but nearly 20%.
- The onboarding subject "Almost Done..." performed about 10% better than "Activate your account"
- Having the video auto play converted visitors almost 50% more often than them having to click the play button (it may be evil, but it makes money).
- Having the video be an animation vs just a spokesperson talking into a webcam converted 5% better with the same script (+2% better with just reading the text from the page).
We had a large audience after we launched our first product free of charge for a limited time (linklicious.co -- formerly linklicious.me -- formerly lts.me), then used that momentum to built out a new product every month or so for the next couple of years. It was a great job in a terrible industry. 80% of my job was inventing new ideas.