UX feedback is usually retrieved via user interviews.

I personally have never heard from a product person „Let’s A/B test whether this is delightful“. And I think that’s because delightfulness or satisfaction is impossible to quantify in A/B tests. You only get to measure things like engagement, signup rates, retention etc. - cold hard taps on the screen, and no more.

And I must say that I‘m glad that, right now, apps can’t just scan my face (or cortisol levels, or pheromones or…) for emotional clues while I read their pesky push notifications that want to coax me back into their daily active user base.

This is not really true. You can pop up surveys in the app or on the site. They can be super non-intrusive. For small effects and large user bases, you simply cannot get useful information from UX studies.

> Your perspective is extremely short-sighted.

it's the perspective of the normal users.

every time i'm using a website and it does not behave exactly the same than for other people or I notice some AB testing, in my head it goes "who the fuck these people think they are?". The computing experience must be consistent and repeatable. If I wanted something that can change depending on the current position of the stars I'd ask another human, not a computer.

A/B testing can result in this type of behaviour but that's just poor A/B testing.

"You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong."

How many tech startup patterns fit that? That's a sign that either the pattern does not generalize well or it's snake oil.

I don't think so. A/B testing is just not a cure-all and alternative for good design, that is all. I mean, if the pattern is 'don't bother thinking too much about ux, just A/B test everything in PROD to death', then yeah, you are right.

However, there are many products which allow users to enter into a beta or test group, where they are the willing subjects to their experiments (in exchange for the latest shiny new stuff). This has the aspect of consent, leaving the 'stable ring' free of such variability. The fact that google and many startups are not using such consent and offering stability, doesn't mean it can't be done or isn't done.

> "You're doing it, but in some subtle but very important way that's not at all obvious to you, you're doing it wrong."

Doesn't that exact statement describe how Agile (or many other concepts, really) is used in a lot of companies? There's nothing wrong with it in principle, but practice is all over the place.

I think of it as validating and improving imperfect user assumptions.

This reads exactly like parody corporatespeak jargon.

Clearly it's the users who are wrong with their stupid assumptions and we will correct them with our mandatory weekly update.

Assumptions about the users, not assumptions made by the users.

I never seen an A/B test where the goal was not to maximise profit for the company.

Have you ever seen "you've already bought this" on Amazon? Highly tested, reduced sales. Overruled by judgement. Stayed on the site. AB test results don't mean you leave your judgement behind.

Maybe. I find that very useful when I'm trying to buy the same thing again. It acts as a trust marker.

I do wish they kept better historic purchase data. I assume it's deleted absurdly quickly for compliance?

Huh, Amazon keeps years of order history for me and you can't even remove it, just "hide" orders which will then still be available just with an extra step. And I'm in the EU so no idea where else they would have more compliance problems with that.

Isn't that any goal of an organization? Everything ladders up to the success of the company. Especially if you're a corporation where you have fiduciary responsibility to the shareholders.

AB testing is beyond UI. It could be AB testing algorithms for recommendations or search results. When i worked for a CPG company, we would AB test the sample sizes in the boxes. To answer the question what is right amount of product for a sample. We would test shipping to speed, was it worth the upgrade to expedited shipping in terms of LTV to justify the extra cost.

What would an A/B test where the goal was not to maximize profit even look like? The very act of creating an A/B test is because the worker wants to improve something in search of higher profits.

Over what time horizon? You should optimize short term metrics (for feasibility) that are (as best you can approximate) the cause of long term profits. The profits part is what makes it the business, using metrics for decisions are what makes it an A/B test. I think where people run into trouble is when they are optimizing only for this quarter’s revenue directly because long term value is too hard to measure.

Oh yeah? I've never seen a marketing department with a decent understanding of microeconomics. Most in fact are trying to maximize budget ie. expense.

That's probably intentional and a sign that it was a well designed A/B test :)

This comment is way too broad and cynical.

It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.

"Reality is complicated" is often a way to rationalize "I'm going to do it how I've always done it, regardless of how it looks."

> This comment is way too broad and cynical

Or, as anyone who has been on the internet since the 90s will tell you, bang on the damn money.

What would the other advantages of A|B testing be for a corporation instead of widening the funnel, increasting lifetime revenue or other such bottom-line focused goals?

Assuming you aren’t running a charity, increasing the bottom line is always your goal. The difference between a company that burns user goodwill in the process, and one that doesn’t, is the difference between short-term and long-term thinking.

The downside with hired testers is that they are unlikely to be a statistically representative sample of your target market. I don’t think this is actually a problem for most startups or new products being launched by major players, since a lot of UX issues early in product development will be obvious to anyone who isn’t the original developers, and the number of actual users you have won’t be big enough to serve as a statistically representative sample of your target market anyway.

you can genuinely find out what message makes more sense.

explaining what your product does is hard when you’re the product expert. A/b testing ad copy helps folks parse what your product does.

If your product is actually good then having more people understand what it is, is also good.

And all of that amounts to what if no one decides to cough up money to use your product?

Let me be even broader and more cynical: everything a company does is ultimately to maximize profit.

Seems like saying no one is actually altruistic because they get good feelings from helping people.

Well, then, the reason why people hate A/B testing is because everyone does it wrong. People can write fast software in Python too but the reason it’s known as being a slow language is because it invariably gets used differently.

A/B testing is a tool, and as Deming said, the aim defines the system. In your definition of Good, you are defining the aim.

I've been in a similar situation, where I created a relatively sophisticated A/* testing and control system. My idea of good use of the system ended up being very different from how the team employing the system thought about it.

I believe that is part of the point of the post, that unintended, and even unimagined side effects plague even the best of ideas.

Do people honestly think that Apple doesn't do A/B testing?

Apple unequivocally does not do active A/B testing on their users by changing applications or the operating system out from under them.

It's almost like they use a dedicated focus/beta testing group or something, instead of making all their users join in.

I can't think of an Apple engineer who would make such a definitive statement on behalf of the company, in public. I also can't think of anyone working there who would have definitive knowledge across all apps and software, because disclosure would prevent such knowledge.

Apple introduced A/B testing to Testflight in 2017 and more A/B testing to the App store this year.

Source: I worked at Apple, but not in software.

Great, but what exactly does that mean for this conversation and how users percieve Apple products?

Yep- it does this in betas, but not in production apps.

You're missing GP's point. This is their experience with A/B testing as practiced. It's not about whether A/B testing could theoretically be great.

I’m not sure this addresses the core criticism of using your users as a testbed.

A/B testing is always going to be distracting to your users though.

Sometimes they'll be in A, and sometimes in B.

The button moves or disappears or appears.

Your user does not get an experience they can rely on

> Your perspective is extremely short-sighted

no, yours is. if some company wants to do some testing, they SHOULD PAY users for that. A/B testing is just exploiting users to get free testing.