Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive more value from the product. Bad A/B testing tries to make things more discoverable, where discoverability is often just noise and distractions. Good A/B testing ensures that the money machine, as you put it, pays its dues to users by making the product experience delightful.
I personally have never heard from a product person „Let’s A/B test whether this is delightful“. And I think that’s because delightfulness or satisfaction is impossible to quantify in A/B tests. You only get to measure things like engagement, signup rates, retention etc. - cold hard taps on the screen, and no more.
And I must say that I‘m glad that, right now, apps can’t just scan my face (or cortisol levels, or pheromones or…) for emotional clues while I read their pesky push notifications that want to coax me back into their daily active user base.
it's the perspective of the normal users.
every time i'm using a website and it does not behave exactly the same than for other people or I notice some AB testing, in my head it goes "who the fuck these people think they are?". The computing experience must be consistent and repeatable. If I wanted something that can change depending on the current position of the stars I'd ask another human, not a computer.
A/B testing can result in this type of behaviour but that's just poor A/B testing.
How many tech startup patterns fit that? That's a sign that either the pattern does not generalize well or it's snake oil.
However, there are many products which allow users to enter into a beta or test group, where they are the willing subjects to their experiments (in exchange for the latest shiny new stuff). This has the aspect of consent, leaving the 'stable ring' free of such variability. The fact that google and many startups are not using such consent and offering stability, doesn't mean it can't be done or isn't done.
Doesn't that exact statement describe how Agile (or many other concepts, really) is used in a lot of companies? There's nothing wrong with it in principle, but practice is all over the place.
I do wish they kept better historic purchase data. I assume it's deleted absurdly quickly for compliance?
AB testing is beyond UI. It could be AB testing algorithms for recommendations or search results. When i worked for a CPG company, we would AB test the sample sizes in the boxes. To answer the question what is right amount of product for a sample. We would test shipping to speed, was it worth the upgrade to expedited shipping in terms of LTV to justify the extra cost.
It reminds me of the sentiment I sometimes hear from my teenage children before I explain to them that the world isn't a Bernie Sander/Reddit gotcha soundbite and that reality is complicated.
Or, as anyone who has been on the internet since the 90s will tell you, bang on the damn money.
The downside with hired testers is that they are unlikely to be a statistically representative sample of your target market. I don’t think this is actually a problem for most startups or new products being launched by major players, since a lot of UX issues early in product development will be obvious to anyone who isn’t the original developers, and the number of actual users you have won’t be big enough to serve as a statistically representative sample of your target market anyway.
explaining what your product does is hard when you’re the product expert. A/b testing ad copy helps folks parse what your product does.
If your product is actually good then having more people understand what it is, is also good.
I've been in a similar situation, where I created a relatively sophisticated A/* testing and control system. My idea of good use of the system ended up being very different from how the team employing the system thought about it.
I believe that is part of the point of the post, that unintended, and even unimagined side effects plague even the best of ideas.
Apple introduced A/B testing to Testflight in 2017 and more A/B testing to the App store this year.
Source: I worked at Apple, but not in software.
Sometimes they'll be in A, and sometimes in B.
The button moves or disappears or appears.
Your user does not get an experience they can rely on
no, yours is. if some company wants to do some testing, they SHOULD PAY users for that. A/B testing is just exploiting users to get free testing.
This shit drives my parents insane. Me too, when I have to help them. I've had to spend tens of seconds looking at a major screen in the phone app, of all things, to figure out WTF I'm looking at so I could help them figure out what was up. Re-arranged every update (or new phone) for absolutely no reason, terrible affordances, poor use of their own design language. Ugh.
I'd get them on iOS but they need larger screens and the $400 small iPhones (what I have) are already more expensive than they think a phone "should" cost, so they keep buying $200 Android phones about once a year (hoping the next one will be better) and not being able to use them because the UI is garbage.
Before I could give her the freshly grandparent-proofed device, said video calling app upgraded on my parents' PC first and changed literally every single element of the UI beyond recognition. To someone the age of my grandma, that would be literally like bricking the device remotely, because none of the buttons would look the same, and she would not be able to work out how to use the new interface.
STOP CHANGING THINGS! Even if the new UI is better (debatable), some people just like or rely on a particular layout to operate the device or app. Don't rearrange without giving a ~permanent setting to use the old layout.
The fact that we're 15 years into smartphones being popular and that phones & computerized address books were basically fully-solved interfaces long before those took off, and they're still fucking around with phone app UI in big ways, is a sign of some kind of institutional or industry-market failure to me. Or both. Allowed, I suppose, because Google's market position (i.e. monopoly across several markets) both gives them the surplus profit (i.e. rents) to fuck around and waste money on harmful to users crap like phone app redesigns, and insulates them from any actual threat to their profit due to those bad choices.
When we rename or remove a function from an API, that's a breaking change, any dependent software will no longer work unless it's modified to take that change into account.
Similarly, when we move, rename, or remove a UI element, keyboard shortcut, or visual affordance, that should be considered a "breaking change" for our dependents, the humans on the other side of the screen. And in the same way, we should avoid making such changes unless the long-term benefit of doing so outweighs the short-term cost. Plus, users will know that moving from 7.x.x of your software to 8.x.x will require them to relearn some aspect of it.
At least on iPad/iPhone you can set the apps to access Google mail, etc, which doesn't change as often, but still too often.
It's kind of mind boggling they'd decide to do that - the replacement they direct me to (Google Chat) doesn't even have feature parity so I just dropped them and moved my social circle using Hangouts to a different app (since at this point they all faced the same problem and we decided on a different platform).
I'm really curious how the A/B testing for this went down - Google is willingly throwing customers away because somebody wants to pump numbers for a new app that is objectively worse than the old one.
At this point Google Maps is the only product that is keeping me with them, but even that one is beginning to wear thin.
Which features are you missing?
It's a complete pita and the family-acceptable-factor is low.
The point is of course to make it annoying to manually update apps and enable auto update. I have been burned too many times with a auto update so I refuse.
This wasn't enough, they really want to force me enable auto updates to the point of the update section of the app having 50% of the visible space on my big screen being covered with a message to enable auto update over WiFi. [0]
Whoever is doing this at Google... Stop. Just stop. It is cringe.
Google seems to be really good at making developer tools like Borg and Blaze - however, I think that as an organization they have some deficit that makes them not responsible enough to develop user-facing software (like, uh, an operating system).
Maybe Google would be better as a B2B company.
In many cases the hardware was so poor it was hard to make a call due to the touchscreen.
Since the primary thing I want the phone to do is make a call I switched to the “it just works” camp and haven’t regretted it.
Except getting photos off the phone. Until I realised the best tool for that is … Ubuntu!
Why can't we have nice things?
If it's something one-and-done (like different permutations of a signup flow to see what is easier for users), then I don't see the harm in it.
You just described doing business in todays world.
Being a bit more generous towards A/B testing, I would make a counter point: _not_ doing any kind of user testing, of which large scale automated A/B tests are just a subset, means you don't respect your users. Because it means you just assume you know what their experience is like, or worse: you don't even care about it and bother to learn something.
Your complaint seems to be more about the scale and aspect of automation honestly, and continuity of the services, which is a valid complaint against Google but not about A/B testing in general.
A/B tests are not the only, or even the best, way of collecting user feedback.
I was just pontif- er, talking about this to someone, a couple of days ago.
I love the users of my products. Most of my products are free, and are carefully-crafted, highly-polished, complete deliverables, and I fret over how they are used -even if by a tiny number of end users-, like a nervous hen. I do what I do, out of love for the craft, and out of a genuine desire to make people's lives easier, through the technology I have at my disposal.
It is my belief that most tech companies despise their user base. Users are little more than cattle, to be fattened and slaughtered. "Caring about the user" means optimizing for "engagement," or keeping them trapped within their own ecosystem. John Oliver did a rant about this, recently[0]. It has nothing to do with actually caring about the user, or solving their problem. It is about harvesting users.
In fact, my discussion about this, came about, because someone wanted to keep users inside the app I'm writing, as opposed to linking them to a more familiar app, on their phone (for the record, it was for videoconferencing). Linking is a "no-brainer," as I can link out to dozens of installed apps, using the simple URL scheme method, built into iOS[1], and "keeping them in the app," would have required several months of extra work, polluting the app with megabytes of junk code, because I'd need to use SDKs, and also kill the ability to easily scale to add new clients (contrary to popular belief, Zoom is not the only videoconferencing option). It would also have possibly put us on the hook, legally, for what happened in those videoconferences.
[0] https://youtu.be/jXf04bhcjbg?t=638
[1] https://developer.apple.com/documentation/xcode/defining-a-c...
A couple years ago, after being an Android fan for the better part of a decade, I finally bought myself an iPhone and pried myself away from Google's ecosystem wherever I could. And Apple didn't even need to do any work for me to make this decision. It was the years of abuse from Google that you experience when you decide to use a Google product or service. And a big part of that was the constant A/B/C/D/E/F testing. I never felt like I was using a complete product, everything felt like a constant beta that could be changed or rearranged at any point, and I was just doing free testing work for them while they harvest all my data.
Every app update was a risk of the app rearranging itself, or features appearing/disappearing. Eventually it didn't even come from app updates in the Play Store, and new interfaces would just appear one day when a server somewhere marked your account as being in the group that gets the new UI. This app that you were familiar with could at any point be rearranged when you open it on any given day. Then maybe a week later you open it and it's back to how it was before. A button you thought was here suddenly isn't, and you question whether something actually changed or if you're losing your mind. It's a subtle gaslighting that eventually I couldn't stand any more.
To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. That's not to say a company like Apple is creating products out of the goodness of their heart, but at least it feels like it was developed by humans who made an opinionated call as to what they thought was the right design decision, and what they would want to use. And in my 2 years of owning an iPhone, I've never opened my reminders app to find out that it's completely unrecognizable, or my messages app has been renamed or rethemed for the umpteenth time.