Statistical significance grows roughly like the square root of the number of samples. Moving from 2-sigma (95%) to the physics gold-standard of 5-sigma requires drastically more data in almost all cases.
Selecting a measurement's uncertainty is something which should be carefully considered. Sometimes you only care about something to 10%, sometimes a 1-in-a-million part failure kills someone's Mom.
If you're doing lots of A/B testing, where trials penalties add up, it might be worth looking into the way that LIGO handles False Alarm Rates. They have to contend with a lot of non-Gaussian noise/glitches.
No, no, no. You are confusing the growth of the standard deviation (which does grow like the square root of the number of samples) with the increase in certainty as you add standard deviations. That falls off like e^(-O(t^2)) where t is the number of samples. This literally falls off faster than exponential.
What does this mean in the real world? In a standard 2-tailed test you get to 95% confidence at 1.96 standard deviations, 99% confidence at 2.58 standard deviations, and 99.9% confidence at 3.29 standard deviations. These numbers are all a long ways away from 5 standard deviations.
Let's flip that around and take 95% confidence as your base. If you are measuring a real difference, then on average 99% confidence requires a test to get 32% more data, and 99.9% confidence requires a test to get 68% more data. Depending on your business, the number of samples that you get are often proportional to the time it takes to run the test. If making errors with x% of your company involves significant dollar figures, the cost of running all of your tests to higher confidence tends to be much, much less than the cost of one mistake.
That is why I say that if the cost of collecting more data is not prohibitive, you shouldn't be satisfied with 95% confidence.
Viewed from my perspective, if you'd like to go from 2-sigma (95%) to 3.29-sigma, you'd need (3.29^2)/(2^2)=2.7 times the amount of data used to get the 2-sigma result, or 170% more samples.
It looks like you've reached your conclusion that I'd need 68% more data to reach 99.9% by taking 3.29/1.95=1.68. I believe that this is in error. Uncertainty (in standard deviation) decreases like 1/\sqrt(N), not 1/N.
\sqrt(N) has driven me to depression more than once.
However consider this. To go from 95% to 99% confidence takes 73% more data collection. So for 73% more data, you get 5x fewer mistakes.
To go from 95% to 99.9% confidence takes 182% more data. So for less than 3x the data, you get 50 times the confidence.
My point remains. Confidence improves very, very rapidly.
As the derivative of \sqrt(N) is 1/2*1/\sqrt(N), your first measurement teaches you the most. Every measurement teaches you less than the last. In general, we measure as much as we must, double the size of the dataset as a consistency check, and move on. The allocation of time is one of the most important decisions of an experimenter.
I've seen a number of businesses who have a current body of active users, and this does not change that fast. So when they run an A/B test, before long their active users are all in it, and before too much longer those of their active users who would have done X will have done X, and data stops piling up. In that case there is a natural amount of data to collect, and you've got to stop at that point and do the best you can.
Businesses are as alike as snowflakes - I am happy to talk about generalities but in the end you have to know what your business looks like and customize to that.
There is no easy answer in statistics!
When you make confidence tests on a continuous process, you will hit this 1 day in 100 on average for 100 days of testing at 99%. If you're doing multiple AB tests per day, well...
EDIT: And another important issue: if you are testing on Valentines day and the pink color/A test is beating out the orange color/B test, then this result may only happen on that one day per year. This applies to hot news topics and numerous other fads as well.
Also make sure your tests really are independent - if you do A/B testing off of fixed IP address hashing and have a small user base, by random chance you may have your top customer always being in 'group A' and giving group A a higher than expected base average - an extra bonus click for every group A test may skew your statistical tests as they assume independence.
If I'm running 100 A/B tests a year, 50% of which produce a 5% improvement in my conversion rate, who cares if I get one of them wrong and drop 5% instead? I'll still have improved my conversions by 1100% over the course of the year.
(Obviously, I'm not taking into account local maxima here - but any statistical misstep will only temporarily roll your testing down the hill, and subsequent tests will push back up to the local maximum anyway.)
Note that in the real world, after a while you exhaust the easy 5% wins. Then you wind up chasing a lot of 1% and 2% marginal wins, and those tests take a lot longer to run. But even so, if your company is pulling down, say a million per month, a 2% drop in business from getting a 1% test wrong probably exceeds the costs of all of the tests you run per year. So unless extending tests is prohibitively expensive, or there are significant opportunity tests from not being able to run the tests you want, you should go to higher confidence.
In my experience, the primary limiter of how many A/B tests you can run is available traffic.
Let's assume that I'm running the 2% tests you mention. I can either run 60 2% tests at 95% confidence, or (approximately) 30 2% tests at 99.9% confidence, if the math posted elsewhere in this thread is right, over a year, say - that's how much traffic I have. Let's once again assume a third of those pan out.
I'm still not seeing why I'd prefer 10 2% wins (from the 99.9% approach) to 20 2% wins plus 3 2% losses (from the 95% approach). Yes, there are more errors, but overall I end up with a 40% improvement as opposed to a 24% improvement.
Can you explain?
And coming up with and implementing likely ideas is a challenge.
Now do you understand?
On the second point - this may be an industry difference, but I've never really had any problem coming up with things to A/B test on a landing page, for example. Just off the top of my head, you could test:
- Headlines - at least 5 of them.
- Headline and subhead vs no subhead.
- Font face.
- Color scheme (overall)
- Images - probably want to test 6-10 of them.
- Call To Action (CTA) text.
- Call To Action button vs link
- CTA placement
- Multiple vs single CTAs.
- Long copy vs short.
- Layout. Image on left, right or top? More than one image? Etc. On the average LP I could come up with 10 possible layouts before pausing for breath.
- Testimonials. PLacement, which ones, how long, picture vs audio vs video vs text.
- Video vs image-and-text.
- Ugly vs good-looking.
- Text highlighting, bolding etc - yellow highlighter style vs bold and italic vs nothing.
- Other social proofing elements - which media icons to use, where to place them, etc.
That's at least 50 A/B tests right there, on a single LP. And all of those elements have been shown in one test or another to affect conversion rates.
You need to be careful if you believe that there is reason to believe that tests will interact. For instance if you're testing different font colors, and different background colors, the possibility of red text on a red background would be unfair to both tests. But in general if you avoid the obvious stuff, you can do things in parallel. (If you have enough traffic you can analyze for interaction effects, but don't plan on doing that unless you know that you have enough traffic to actually follow the plan.)
The first paragraph is talking about random interaction. So, for instance, version A of test 1 was really good, and version B of test 2 got more A's from test 1 than version A of test 2 did. This gives version B a random boost. As long as things are random, it is OK to completely ignore this type of random interaction from the fact that you are running multiple tests on the same traffic.
The second paragraph is talking about non-random interactions. People who are in version A of test 1 and also in version B of test 2 get a horrible interaction that hurts both of those. If you have reason to believe that you have causal interactions like this, you can't ignore it but have to think things through carefully.
His math is right, but the logic misses a basic fact. In A/B testing nobody cares if you draw a conclusion when there is really no difference, because that is a bad decision that costs no money. What people properly should care about is drawing the wrong conclusion when there is a real difference. But if there is a significant difference, only for small samples sizes is there a realistic chance of drawing a wrong conclusion, and after that the only question is whether the bias has been strong enough to make the statistical conclusion right.
He also is using 95% confidence as a cut-off. Don't do that. You don't need much more data to massively increase the confidence level, and so if the cost of collecting it is not prohibitive you absolutely should go ahead and do that. Particularly if you're tracking multiple statistics. If you test regularly those 5% chances of error add up fast.