I'm surprised no one on the team was familiar with multivariate analysis. That's the wrong way to say it. Let me rephrase more carefully...

I'm surprised no one used more sophisticated analysis techniques to allow an experiment that changes many explanatory factors simultaneously and estimates the effects of not only each factor independently but also their interactions. It's relatively straightforward statistics and would have been taught to clever undergrads or first-year grad students who specialized in math, economics, machine learning, industrial engineering, etc.

There were stats Ph.Ds on the team that were certainly familiar with it, and I knew at least one former Amazonian manager that held that we were too afraid of having multiple experiments running at once and too afraid of confounding factors. (Amazon's A/B testing is much more traditional split-testing, where they show one variant to half the population and one to another half and then pick the best one.)

The biggest problem is that many of the folks involved had encountered very surprising and counter-intuitive interactions between features, and because of that, they didn't trust the textbook models. For example, at one point we had a very convenient embedded-Python experimentation framework that let us rapidly try things out. Our main data-scientist on the project decreed that we were not to use it, because a number of the externalities that framework introduced - higher latency, and potential error conditions - could confound the metrics in ways that were not well-understood at the time. Data is useless unless you can trust that it actually represents reality.

Stats textbooks all make certain assumptions about reality when they present the models, and if reality doesn't conform to the assumptions, the models won't conform to reality. Certain factors were well-studied enough that we could build useful models from them - that was the point of all the experimentation around color, after all. (I wonder if those conclusions still hold, anyway: data has an expiration date, and I recall personally running an experiment that directly contradicted an experiment Marissa Mayer had run 6-8 years previously.) But that requires a lot of hypothesis testing and verification: it's never a matter of "well, the model says so, therefore it must be", but rather "well, the model says so, let's run an experiment to see if the model's predictions line up with reality, and if they do, we have a pretty strong - but not ironclad - indication that we can use it for future calculations."

tl;dr: Largest and best-funded social science program I've ever seen.

> tl;dr: Largest and best-funded social science program I've ever seen.

I don't want to sound like a hater and I'm very well aware of the fact that some design choices might have affected the company's revenues, but all this discussion involving very smart people discussing what, from a very "outsider-ish" perspective, seem like trivial issues (the color of the letter G, for example) reminds me of Herman Hesse's "Glass Bead Game". (https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot)

>Largest and best-funded social science program I've ever seen.

You should check out Facebook. There emotional manipulation studies are horrifying, but they certainly have access to many more person hours of time and collected data than Google does on end users.

I understand the urge to try to maximize on stuff like this, but in the end it loses sight of what's most important: the quality of the results. And sometimes I feel that a lot of valuable engineering time and effort is wasted on making things prettier and better monetizing in the short term when in the longer term the better search results are what would really move the needle.

And most A/B testing focuses on the very short time effects of the change (and change all by itself, even change for the worse can have a short term positive effect just because something is new). On another note: A/B testing has another limitation, which is that you're optimizing for the bulk, please one person more and you more or less automatically annoy another.

There are plenty of talented people working on ranking and webspam too.

There's a limit to how parallelizable those problems are. At some point, more people working on an algorithm doesn't make it faster or better, it just makes it more confusing and worse. Mythical Man Month applies as much to Google as it did to IBM. And so you might as well put people onto optimizing the rest of the page as well, because they'll do more good there than having yet another cook in the ranking algorithm kitchen. (I'd argue that there are too many people working at Google in general...I left, so I'm no longer contributing to that problem. But then, there were probably too many people when I joined, so I contributed to it for 5 years.)

The worst limitation of A/B testing is that it's not producing anything better than a local maxima. Nature does indeed work like that, but it took millions of years along with random mutations and we don't have that much time. And if in our thinking process we would have relied on optimization algorithms favoring the local maxima, then Jazz wouldn't have been produced, amongst others.

Can you offer any perspective on the whole sometimes-named "mystery meat" UI that's been rolled particularly but not only into mobile design over the past half decade?

Further, more and more of this seems to have been "backported" into traditional desktop presentation design.

While I will experiment around until I more or less "get it" (although I may miss finding some useful features for months), it particularly frustrates people like my parents. Mom is somewhat spatially challenged (although plenty bright in other ways), and I've had numerous calls with her where the controls for what she wants to accomplish have changed and simply bewilder her. AND, given her spatial orientation problems, sometimes we will have to have the same conversation several times -- because the rather arbitrary-appearing icons and patterns don't immediately "sink in".

P.S. I mean this as a legitimate question, if it fits the parent's -- or others' -- field of experience. It seems that a lot of this design is expected to simply permeate and be picked up by osmosis. Is that really the case?

I'm curious what you mean by "mystery meat"? I assume it's the use of icons and drawers to trigger functionality rather than labels and links?

That's driven by screen size - there's simply not enough screen real estate on a phone to label every button with words. It does have a legit discoverability problem, which most folks in UX acknowledge but don't know how to fix, given other constraints of the media. I've seen a bunch of mobile apps with help overlays or tour videos the first time you run them, which can help a lot. It gets back-ported to desktop apps because once people do learn the functionality, it leads to a cleaner, less cluttered UI.

I'll also point out that early PCs had the same problem. A typical PC from the early 1980s had 320x200 resolution, a little less than a smartwatch does today. For most of the 80s and early 90s, the standard was VGA (640x480), slightly smaller than most smartphones today. Icons were really common back then as well, they weren't all that discoverable, but at least people read manuals occasionally. Most folks don't bother now. Menus were another response to that problem, but they don't really work on mobile because the tap target is too small to efficiently hit it (the navigation drawer & hamburger menu is the modern-day equivalent of the menu).

The web - with its inherent discoverability and all of its space for explanatory text - didn't really become popular until 1280x1024 displays became widespread. I remember designing webapps for 800x600 screens; it was painful, and they had discoverability problems too.

Well, a, um, "literal" example was the initial occurrence of what I've just learned is called the "hamburger" icon/menu. Which was "backported" to Google's desktop designs.

A particular instance I recall was changes to the Calendar web page presentation. Mom initially struggled a fair amount with the controls in the left bar for selecting which calendars were displayed. This was in a Google Apps environment with 20-ish people participating.

I remember when screen presentations could be somewhat cryptic -- per your description as well as other instances. But, as you point out, there was usually a manual. Now? Often it seems to be down to guessing and poking around. Or asking friends. But if you are a bit older and not constantly talking over such things with your friends, that means of edification is somewhat starved.

P.S. Thanks for the response. And as I said, I'm curious about the perspective of someone on "the other side". I'm not trying to be unduly critical, here; just, this is an issue I've consistently bumped up against, from my own perspective.

> That's driven by screen size - there's simply not enough screen real estate on a phone to label every button with words. It does have a legit discoverability problem, which most folks in UX acknowledge but don't know how to fix, given other constraints of the media.

Some of the most effective solutions to this I've seen require solutions beyond visual representation. They address the problem through dimensions independent of viewport size and space.

For example, LukeW's "Don't Divert the Train" http://www.lukew.com/ff/entry.asp?1798 moves content from behind navigation and into the user's already-occurring action stream. The button is removed entirely, as is the designer's need to consider how communicative icons are for representing actions.

This requires a broader view of how features relate to each other through a design process that discovers other paths to successful behavior. Approaches like the hamburger menu, in addition to the usability issues you note, solve functional goals but have fallen out of vogue for their ineffectiveness at encouraging specific actions. Label-less icons, while attractive and convenient, suffer similar problems. As getting users successful quickly is generally an important goal (before they've internalized your app's unique visual language), the current trend of putting core functionality behind opaque symbols seems counterproductive.

Another thing I'm now reminded of is, on Android, the... functionalization/clickification of text.

Some text is active, other is not. I don't see any visual cues that distinguish the two. I end up just poking at text, or thinking, "If I were designing this, I'd want more detail or a link to further controls." In the latter case, sometimes I find I was thinking the way the designer/programmer was, and that text does indeed lead somewhere.

For someone who doesn't "think that way", this is more of a challenge.

This also seems to have become more prevalent in desktop / full screen design. Stuff that doesn't look "clicky" is.

I think that was one of my mother's problems with changes to the Google Calendar web page UI.

Interestingly enough, LukeW is now doing product/design for Google.

Design by AB testing, my favorite :(

I worked at a major tech company where every decision needed to be justified by AB testing. Our design was a series of haphazard guesses made on data that depended on a very brittle data pipeline. Needless to say, nobody thought it was a very good product, but the incredible bureaucracy and people behind the approach made it hard to fight against.

It shouldn't be a problem. In science, everything must be tested and proved if you want to know the truth or the effectiveness of an theory. Design is a science, so it can be measured and developed by tests of all kinds, even emotionals.

That's a pretty one-sided statement. One could easily argue the opposite, that good design is an art form, not a science.

Art is a science too. One different thing is that some artists don't care about the facts. Everything can be explained with numbers.

Whereas a design studio with sufficient resources should fare better with a data-driven approach, a sole designer or small team with time and feedback constraints will probably do better with traditional methodology. Traditional methodologies might have certain quack dogmatic rules and processes, but it offers a battle-tested baseline.

> One of our biggest problems was that since we tried to change everything at once, the metrics went haywire

Isn't that the problem with empirical methods in general? They are great for measuring the effect of small simple things (like small changes to a web page), but are much less effective in measuring bigger things (like say, programming language A vs. programming language B).

Absolutely. Business is an interesting field because you derive a fairly large advantage from being correct about reality, but that advantage counts for nothing if you're so slow that somebody else changes the reality while you're figuring it out.

That is a very succinct way of putting it, thank you.