scuba_man_spiff
scuba_man_spiff on Hacker News
The solution mentioned of running a two tailed test would not have solved the problem of a false result the author demonstrated through conducting an A/A test.
According to the image in the article: http://blog.sumall.com/wp-content/uploads/2014/06/optimizely...
The A/A test had: A1: Population: 3920 Conversion: 721 A2: Population: 3999 Conversion: 623
Z-Score: 3.3
2-tailed test signifiance: 99.92%
Looks like the one-tail vs. two tail test doesn't make huge difference in this case.So, maybe a larger sample size would have seen a reversion to the mean, but given the size and high significance that would be unlikely (interesting exercise to try different assumptions to calculate how unlikely, with the most overly generous obviously just being the stated significance).
Yes, the test was only conducted over one day, but if it was the exact same thing being served for both, that shouldn't matter.
If there was a reversion to the mean due to an early spike, we would expect to see the % difference between the two cells narrow as the test kept running. You can see in the chart that the % difference (relative gap between the lines) stays about the same after 8pm on the 9th.
So if it's not the one-tailed test at fault, and it's not the short duration of the test at fault, what is?
Don't know.
I have seen in the past that setup problems are incredibly easy to make w/ a/b testing tools when implementing the tool on your site. I've seen in other tools things like automated traffic from Akamai only going to the default control, or subsets of traffic such as returning visitors excluded from some cells but not others.
Based on those results, I'd be suspicious of something in the tool setup being amiss.
Standard statistical tests used in a/b testing are based on one check. If someone is checking repeatedly on a test until they get a 'significant' result, your chance of getting a getting a false positive is many X the stated significance.
Best practice - set a pre-defined end, and one or two defined early check-in points where only make an early call if result is overwhelmingly significant or if the business has fallen off a cliff.
The good thing though is that there's a tipping point, where once ~1/3 or so of people fully grok how to evaluate and understand the metrics they're looking at, those who do understand start helping (or calling out) those that don't, lifting competence throughout the organization.
http://www.amazon.com/Machine-Learning-Python-Techniques-Pre...
The main thing to understand though is that machine learning is a big topic, and you aren't going to be able to become an expert in two months.
Narrow down to a specific area, or type of problem, and focus on learning techniques and tools for that.
My guess is that there's something your working on or want to work on which is why want to learn. If that's the case, I'd recommend that read up a bit to give yourself a good understanding of the different kinds of problems out there (classification, prediction, anomaly detection, etc...), and different classes of tools available, and then pick a simple real world problem to try to tackle that is similar.
The best way to really learn is going to be getting hands on with a project and suffering through after you've read up a bit to understand the basics. Then when you hit something can't wrap your head around, search and read articles (or talk to someone with experience and expertise) until it clicks and you can proceed on working through.
By the end you'll have a good grasp of at least one technique, and be in a great place to keep learning more.
If anyone's tallying