That's exactly my thinking as well. To give another example:
This week I tried to use an iPad and an Apple Pencil for day-to-day note taking. If you handwrite 100% plain English prose like, The quick brown fox jumps over the lazy dog, it instantly converts it to text and corrects any typos. For simple stuff, it seems perfect.
But it's insufferable once you start on real work. It doesn't understand a lot of punctuation, tries to correct foreign words (likes names and locations) I mix in with English, tries to spell correct serial numbers (N07 becomes NOT). Having to go back and correct each conversion error turned out to be much slower -- and frustrating. I went back to a keyboard.
The iPhone camera is usually excellent, but there are some situations (for me it's autumn forests and funnily enough Disneyland at night) in which the auto white balance fails miserably. On my DSLM it's a simple question of just using manual white balance, but on the iPhone the only way to do it with the stock camera app is to do in post.
On that note, the Photos Mac app has a neat feature where you can fix white balance by calibrating on a neutral grey area, but that feature is missing on the mobile apps which just have a color temperature slider which does not give as good results.
This was very jarring at first, but over time I've actually gotten used to the gestures and they feel more consistent than various makers of phones having the software buttons in different positions. It's almost like muscle memory at this point.
If you don't like gestures, however, you should be able to go to the system settings and enable the old style navigation. If they had removed it entirely, then I'd probably be more critical (a la Windows 11 vertical taskbar).
Losing those buttons was a pretty serious usability hit that has never been made up for with touch screens.
The new version ran on Windows, had a lot more features for sure, but suddenly staff was entering data into a field, grabbed the mouse to move to the next field, entered data, and so on. The process became way slower. So I asked why they don't just ignore the mouse and keep using tab. They were surprised this even works, because it was never mentioned by the folks who trained them on the new software. It was all about how modern and slick it looks and how it has so many more features.
So then staff tried to use the tab key again, but got frustrated very quickly since the software was smart enough so enable and disable controls in the entry form depending on selections you made in earlier fields. So tabbing through the form would require different number of key presses all the time, making it hard to build muscle memory. I think they raised their complaints to management but whether that feedback ever reached the software vendor I have no idea. I guess they did get faster again over time because any new system takes time to get used to, but I'd be very surprised if it ever matched the ancient, simple system again.
We used for years (decades) a DOS based tool, where in a "conventional" 80x25 mask there was enough space for all the data we had to input (navigating with TAB/Enter, etc.) before passing to the next (with F8).
On the new software the same data (in windows, on a modern screen) goes inside a teeny-tiny window that has not enough fields, so it is spread on three tabs and this breaks the flow.
The same happens a lot on forms on the web, fill one or two fields, click next, repeat.
If this is indeed the case, I'm just shocked that developers and companies haven't figured it out. I suppose a lot of the time the performance impact simply isn't measured in their kpis
My other hypothesis is that once you move away from an orderly website architecture, organizations and devs are just as lost as the users and no one has an idea where information should reside. Without holistic architecture or vision, the best that people can do is throw a third or fourth hamburger on the home page, add a new page 10 links deep, or not linked at all and pray that a search engine will solve the problem.
I wouldn’t want the take-away point from your experience there to be over-generalised.
I replied carefully, not wanting to nitpick. But I think the distinction is important.
Without that, it can be disastrous.
I agree with you that having users take a single sip is also a problem for construct validity. But it’s not clear from your post how this would this would happen in a properly run AB test.
Are you thinking of the case in which we ship a change that confers short term wins in user spend or churn, but comes at a long term cost that isn’t visible within the window of the test? This is a common problem, but it’s not clear to me that it’s the same problem Pepsi had. Here the problem is with choice of response variable, whereas Pepsi’s problem is choice of treatment.
It's about what you actually measure versus what you think you are measuring. In this case, the can satisfaction vs versus the sip.
Short term spend or churn could be examples. For an information serving website, examples might be mistaking time on the site or pages viewed for user satisfaction and finding the information they want.
I see the same problem with user tests. People almost always pick the simpler of two options, because at the time they have no reason to pick anything else, and that looks easier. But then you start trying to get actual work done, and the Fisher Priced interface is too simplified.
And I know it's not just you or me that prefer information dense websites. Look at how well-known McMaster's site is, among people who actually need to use it to get work done.
Even in general public terms, things have gone too far. I set up a lot of iPads for elderly folks for use in assistive communication, and I have to go out of my way to use ones that have buttons. The modern ones without a home button are just too complicated for a whole swath of our population, and I am not exaggerating.