I've personally run tests and seen tests achieve statistically significant (and very large) results that weren't merely fixing bugs or moving buttons within reach of a user. It could be things like adding a modal, changing copy (yes, literally just adding or changing words on the page), removing content, changing an image, or reordering funnel steps.
Think about that: by merely changing some sentences, your website could be 10% more successful next month. But you won't know whether it's working without a/b testing.
You don't just create tests randomly, but based off of an understanding of the product, goals, signup funnel, etc. which may involve talking to users, but ultimately verifying the change is for the better through a/b testing.
I would agree that there are a number of people running a/b tests that don't understand the math or know what they are doing, but that isn't an indictment of a/b testing.
You should read the article. It explained very well why a) statistical significance is meaningless when split testing websites (you are guaranteed to archive significance if you run the test long enough; the math behind significance testing doesn't work like assumed by Optimizely et.al.) and b) most tools on the market are inherently unable to prove effect size by the way tests are designed.
Just do this short experiment yourself: Do an a/b test with identical versions of you site. Spoiler: With 100% certainty, you'll see both significant results and a sizable effect size difference.
statistical significance is meaningless when split
testing websites (you are guaranteed to archive
significance if you run the test long enough
Wait, what? Not if you use the right test to evaluate your change. This article isn't showing that A/B testing is bunk, it's showing that Optimizely is using an inappropriate test. Do an a/b test with identical versions of you site.
Spoiler: With 100% certainty, you'll see both significant
results and a sizable effect size difference.
The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical. Running your A vs A experiment and evaluating it correctly you'll find results significant at the p < N rate N% of the time. And the longer you run it the lower your estimate of the effect size will be.> Wait, what? Not if you use the right test to evaluate your change.
Sure, but how often to people actually choose the right combination of test, testing tool and estimated effect size for their significance to have actual meaning?
The problem with experiments like your typical website split test is the potentially infinite population. With a big enough N, the probability of reaching significance approaches 1.
> The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical.
Yes, I know. But the magic lies in that "done properly". Have you ever approached a tool in the field that would fail my proposed test? I haven't.
With a big enough N, the probability of reaching
significance approaches 1.
Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%.On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care.
Have you ever approached a tool in the field that
would fail my proposed test?
Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments.(If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)
A common place where people get stuck when A/B testing is in a local minimum or maximum. For example, instead of tweaking button colors someone should instead redesign the whole form. Talking to users can help with this, but users are notoriously bad at not knowing what they really want.
Anyway, on to your comment:
> A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.
I don't know what this means. Can you elaborate? What exactly is "significant instrumentation"?
> That, or actually build a product that users want instead of chasing after pointless metrics.
Pointless metrics? I'm not sure what metrics you measure but visitor engagement and visitor to trial conversion rates are the lifeblood of a software sales and marketing funnel.
When somebody in marketing considers A/B testing as a means to increasing trial and paying customer conversions, they're operating on the basis that their product already provides value. The A/B tests are a means of establishing the best method for communicating that value. Better communication = more trials. More trials = a larger pool of potential customers to convert to paying customers. It's pretty basic.
> Most of the A/B website optimization industry is an elaborate scam
What?? Had you said a number of people in marketing could be in danger of blindly relying on their tools (e.g. Optimizely) and not applying common sense or checking the maths I would have wholeheartedly agreed. In my personal experience with the various websites I market, and given the hundreds of split testing examples I've read from people I do not feel are in on the "scam", I've got to strongly disagree with this point.
There is nothing inherently wrong with A/B testing, and it's basically the only way to verify that anything you do has a positive impact. What's created a hype bubble is articles and platforms saying that minor UI changes, colour of buttons etc. make a real difference to conversion rates. Good tests based on well thought out hypotheses relevant to your actual customers, can and sometimes do help websites find measurable gains. It is not a silver bullet, and no matter how careful you are there is always statistical uncertainty, it won't always help.
In my experience this only works for obvious, glaring flaws, such as "I hate that the phone number field enforces a specific format" or "Why the hell are your password requirements so stringent?"
But a lot of UX is about what happens on the unconscious level. A user will never tell you, "Your bright red background feels too aggressive/hostile" because they simply don't recognize it at the conscious level to be able to verbalize it. The signup or checkout form might make them feel uneasy, but they won't know why.
I find that the best way to optimize websites is to do A/B testing first, and then talk to your users about the subset of the results (which might be the majority) that you find confusing or surprising. Basically, never rely on just one method, because then you will be hamstrung by the shortcomings of that method.
The overall utility of A/B tests is a complex topic and largely depends on the skill of the experimenter. Well-designed experiments are extraordinarily useful. A/B tests are one tool for product development, but they can't replace it.
Many, many people have successfully used A/B Testing. I've personally used it to great effect several times. I certainly don't make decisions purely based on the statistical results, but I find it to be an extremely useful input to the decision making process. All models are flawed; some are useful.
Even talking to users would not help much if one continues to build a faster horse.
Changing color of text is not much improvement, unless the text was unreadable or same color as background.
No amount of data driven can replace what the actual product user says.
But but but telemetry is evil!
Telemetry in my OS* that has access to everything? No.
* That I cannot turn off.
I like Windows and really like VS, I was often literally the only (non-VM) Windows user in a sea of OS X at several offices but Windows 10, the flip-flopping UI, the ads, and the telemetry pushed me to Linux and building my own desktops.
A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.
That, or actually build a product that users want instead of chasing after pointless metrics. I mean, really, you think changing the color of text or a call-out is going to make up for huge deficiencies in your product or make people buy it? The entire premise seems illogical and just doesn't work. The only time I've seen a/b tests truly help was when it accidentally fixed some cross browser issue or moved a button within reach of a user.
Most of the A/B website optimization industry is an elaborate scam, put off on people who don't know any better and are looking for a magic bullet.