I was the first engineer on the first visual redesign that tried to change everything about the page [1]. One of our biggest problems was that since we tried to change everything at once, the metrics went haywire, and we couldn't know if a change was because we introduced a left-nav (okay, actually we kinda could, because we'd experimented with a left-nav on its own beforehand) or because we changed the line spacing or because we cleaned up the logo or because we added icons. There was effectively no way of "debugging" the user; when user behavior varied from expected, we didn't know why.

I've got plenty of other stories from that era, but they aren't really fit for a public forum. Happy to discuss privately. (Kinda ironic when the non-Googler says that to the Googler...)

[1] http://googleblog.blogspot.com/2010/05/spring-metamorphosis-...

I'm surprised no one used more sophisticated analysis techniques to allow an experiment that changes many explanatory factors simultaneously and estimates the effects of not only each factor independently but also their interactions. It's relatively straightforward statistics and would have been taught to clever undergrads or first-year grad students who specialized in math, economics, machine learning, industrial engineering, etc.

The biggest problem is that many of the folks involved had encountered very surprising and counter-intuitive interactions between features, and because of that, they didn't trust the textbook models. For example, at one point we had a very convenient embedded-Python experimentation framework that let us rapidly try things out. Our main data-scientist on the project decreed that we were not to use it, because a number of the externalities that framework introduced - higher latency, and potential error conditions - could confound the metrics in ways that were not well-understood at the time. Data is useless unless you can trust that it actually represents reality.

Stats textbooks all make certain assumptions about reality when they present the models, and if reality doesn't conform to the assumptions, the models won't conform to reality. Certain factors were well-studied enough that we could build useful models from them - that was the point of all the experimentation around color, after all. (I wonder if those conclusions still hold, anyway: data has an expiration date, and I recall personally running an experiment that directly contradicted an experiment Marissa Mayer had run 6-8 years previously.) But that requires a lot of hypothesis testing and verification: it's never a matter of "well, the model says so, therefore it must be", but rather "well, the model says so, let's run an experiment to see if the model's predictions line up with reality, and if they do, we have a pretty strong - but not ironclad - indication that we can use it for future calculations."

tl;dr: Largest and best-funded social science program I've ever seen.

I don't want to sound like a hater and I'm very well aware of the fact that some design choices might have affected the company's revenues, but all this discussion involving very smart people discussing what, from a very "outsider-ish" perspective, seem like trivial issues (the color of the letter G, for example) reminds me of Herman Hesse's "Glass Bead Game". (https://en.wikipedia.org/wiki/The_Glass_Bead_Game#Plot)

You should check out Facebook. There emotional manipulation studies are horrifying, but they certainly have access to many more person hours of time and collected data than Google does on end users.

And most A/B testing focuses on the very short time effects of the change (and change all by itself, even change for the worse can have a short term positive effect just because something is new). On another note: A/B testing has another limitation, which is that you're optimizing for the bulk, please one person more and you more or less automatically annoy another.

There's a limit to how parallelizable those problems are. At some point, more people working on an algorithm doesn't make it faster or better, it just makes it more confusing and worse. Mythical Man Month applies as much to Google as it did to IBM. And so you might as well put people onto optimizing the rest of the page as well, because they'll do more good there than having yet another cook in the ranking algorithm kitchen. (I'd argue that there are too many people working at Google in general...I left, so I'm no longer contributing to that problem. But then, there were probably too many people when I joined, so I contributed to it for 5 years.)

Further, more and more of this seems to have been "backported" into traditional desktop presentation design.

While I will experiment around until I more or less "get it" (although I may miss finding some useful features for months), it particularly frustrates people like my parents. Mom is somewhat spatially challenged (although plenty bright in other ways), and I've had numerous calls with her where the controls for what she wants to accomplish have changed and simply bewilder her. AND, given her spatial orientation problems, sometimes we will have to have the same conversation several times -- because the rather arbitrary-appearing icons and patterns don't immediately "sink in".

P.S. I mean this as a legitimate question, if it fits the parent's -- or others' -- field of experience. It seems that a lot of this design is expected to simply permeate and be picked up by osmosis. Is that really the case?

That's driven by screen size - there's simply not enough screen real estate on a phone to label every button with words. It does have a legit discoverability problem, which most folks in UX acknowledge but don't know how to fix, given other constraints of the media. I've seen a bunch of mobile apps with help overlays or tour videos the first time you run them, which can help a lot. It gets back-ported to desktop apps because once people do learn the functionality, it leads to a cleaner, less cluttered UI.

I'll also point out that early PCs had the same problem. A typical PC from the early 1980s had 320x200 resolution, a little less than a smartwatch does today. For most of the 80s and early 90s, the standard was VGA (640x480), slightly smaller than most smartphones today. Icons were really common back then as well, they weren't all that discoverable, but at least people read manuals occasionally. Most folks don't bother now. Menus were another response to that problem, but they don't really work on mobile because the tap target is too small to efficiently hit it (the navigation drawer & hamburger menu is the modern-day equivalent of the menu).

The web - with its inherent discoverability and all of its space for explanatory text - didn't really become popular until 1280x1024 displays became widespread. I remember designing webapps for 800x600 screens; it was painful, and they had discoverability problems too.

A particular instance I recall was changes to the Calendar web page presentation. Mom initially struggled a fair amount with the controls in the left bar for selecting which calendars were displayed. This was in a Google Apps environment with 20-ish people participating.

I remember when screen presentations could be somewhat cryptic -- per your description as well as other instances. But, as you point out, there was usually a manual. Now? Often it seems to be down to guessing and poking around. Or asking friends. But if you are a bit older and not constantly talking over such things with your friends, that means of edification is somewhat starved.

P.S. Thanks for the response. And as I said, I'm curious about the perspective of someone on "the other side". I'm not trying to be unduly critical, here; just, this is an issue I've consistently bumped up against, from my own perspective.

Some of the most effective solutions to this I've seen require solutions beyond visual representation. They address the problem through dimensions independent of viewport size and space.

For example, LukeW's "Don't Divert the Train" http://www.lukew.com/ff/entry.asp?1798 moves content from behind navigation and into the user's already-occurring action stream. The button is removed entirely, as is the designer's need to consider how communicative icons are for representing actions.

This requires a broader view of how features relate to each other through a design process that discovers other paths to successful behavior. Approaches like the hamburger menu, in addition to the usability issues you note, solve functional goals but have fallen out of vogue for their ineffectiveness at encouraging specific actions. Label-less icons, while attractive and convenient, suffer similar problems. As getting users successful quickly is generally an important goal (before they've internalized your app's unique visual language), the current trend of putting core functionality behind opaque symbols seems counterproductive.

Some text is active, other is not. I don't see any visual cues that distinguish the two. I end up just poking at text, or thinking, "If I were designing this, I'd want more detail or a link to further controls." In the latter case, sometimes I find I was thinking the way the designer/programmer was, and that text does indeed lead somewhere.

For someone who doesn't "think that way", this is more of a challenge.

This also seems to have become more prevalent in desktop / full screen design. Stuff that doesn't look "clicky" is.

I think that was one of my mother's problems with changes to the Google Calendar web page UI.

I worked at a major tech company where every decision needed to be justified by AB testing. Our design was a series of haphazard guesses made on data that depended on a very brittle data pipeline. Needless to say, nobody thought it was a very good product, but the incredible bureaucracy and people behind the approach made it hard to fight against.

Isn't that the problem with empirical methods in general? They are great for measuring the effect of small simple things (like small changes to a web page), but are much less effective in measuring bigger things (like say, programming language A vs. programming language B).

https://news.ycombinator.com/item?id=10153449

I'm just not a fan of extremely flat designs.

https://www.google.ca/?doodle=22175023&hl=en&nord=1

https://www.google.com/favicon.ico?newlogo

(I'm seeing a four-colors capitalize G.)

Thank you for linking it, hadn't reached my interwebs yet.

This is not only applies to caching by ISP, but also includes caching by the website itself as well (some websites put proxy in front of their server to cache rarely changed portion of the site, or even using CDN.).

https://www.google.com/favicon.ico?anyquery

First column, fourth row down: http://2.bp.blogspot.com/_7ZYqYi4xigk/SEnK37orPGI/AAAAAAAAAp...

Honestly, the block letter G on the far right, next to the lollipop, is the icon with the most character.

Actually fits their rebranding, too, but I don't think anyone at Google pays much attention.

Everyone's been seeing a lot of rainbows lately, too.

That's because they were designing a favicon, not doing a rebrand. Using a different font than your logo would make absolutely no sense.

I believe I have successfully refuted that claim.

I know it's weird, but favicons are weird.

Anyways I tried it in IE, including restarting it and it still showed the old lowercase g.

BUT, when you're the size of Google, or Facebook, or Microsoft, Things look a bit different.

Most people are far more conservative than they realise, but also, perversely, adapt quite quickly to change if they experience it day-to-day. This means that if an interface has enough inertia, then a change that performs badly in user testing, can actually quite quickly be accepted by people. (PROVIDED it's actually better).

In the first example, every change to the Facebook UI is met by a barrage of criticism and hatred, but they seem to actually make good decisions, so the changes get absorbed by users, and things continue.

Having said that, Microsoft struggle, because they often make changes that are fundamentally bad decisions, and have to back-down. For example, the Office ribbon took a while to be accepted, but is /generally/ liked nowadays by people, because it was functionally a good change (yes, some people will disagree, but the majority wins on this front). Taking broad user testing results on the ribbon would probably have stopped microsoft from progressing with it. On the other hand, the 'Metro'/Tile UI in windows 8 was an actual usability disaster, and has forced them to revert to a more common pattern until someone can design new metaphors that actually improve usability in windows.

Finding test subjects that aren't used to, and comfortable with, the google font/design styles, to give objective feedback on a change, is really hard.

[ I didn't mention first time round, but I /really/ don't like the new logo, that lowercase g makes me cringe inside every time I see it. ]

The evolutionary, data-driven approach will never yield a Nest thermostat, iPhone, Helvetica, or flag of New Mexico. We have no idea how to even begin to write a fitness function for good design.

https://en.wikipedia.org/wiki/Akzidenz-Grotesk

Formalizing that process and using externally-collected data is just de-black-boxing something that happens in a black-box form in the professional's mind.

You point out the need to pick the right measurement, and I agree, but it could be as simple as "What's the effect on conversion?" It has to be something! You can't just say "We're re-designing because I'm a HMI genius."

Until then, the best known way to design something nice is to hire a talented designer and trust their instincts.

...for now, given today's technology, but I wouldn't bet on it long term. History is full of examples of things we once thought only humans could do.

But once many candidate designs are available to choose from, what better a way is there to unambiguously decide on the best besides testing and measuring?

But at that point, why not just cut out the middleman, forget the tests, and let the artists be artists?

See also: Footnote 3 of http://www.paulgraham.com/colleges.html#f3n

The trick is: there is no global "better" — only 'better', given a specific group of users.

Which users though? Running a million 40 year olds from Iowa through your website flow will generate a widely different result from running a million 13 year olds in Atlanta through the same process.

Google is still stuck in the "one design for everybody in the world" mindset, so they have to be appealing to everybody, which naturally makes them bland.

success metrics you're using

Success metrics! The paradox of these hyper-optimization methods is they largely apply in hyper-trivial situations. Once you are big enough, you just tell people how things will be. They don't get to have a say anymore.

And you can't rely on past data to create new trends because that data doesn't even exist.

Data said that children's books don't sell. But then Harry Potter happened. Data also said that people loved phones with keyboards. Then the iPhone happened.