I have never worked at a company that did user testing or if they did it was always done in a way or interpreted in a way to back up the designer's opinion. I don't think I've once in my entire 40 yr career seen a designer test with users, find out something was bad, and change their design based on the test.
Has any one else?
To be clear I have seen a designer test themselves and re-design but I've never seen them test with users and re-design. I've also seen them change a design, put it out in limited release, then claim "we didn't get any/many complaints so it must be okay" without a thought that the majority of users never complain (either don't know how, can't be bothered, never considered it might be useful, clicked the feedback but that doesn't actually make it back to the designer, combinations of all of the above)
I agree that there is a temptation to not do it, especially for smaller changes, especially if time is tight, especially if you are a smaller company, and all of that is certainly a problem.
But no user testing? Not changing the design? We just recently drastically changed the design of a new product we are developing because it failed initial user testing (with clients of us who came in for user testing using a paper prototype).
What followed were also several rounds of design critique sessions with the revised design (bringing in internal people who had nothing to do with the design and ask them for their constructive feedback, without aiming to find a solution to what they find during that meeting) and we will bring those original clients back in in December and sit them in front of the then working prototype.
We are a small company and I do agree that we do this far too infrequently but from my viewpoint as someone who makes the design I couldn’t even imagine not reacting to people who clearly have trouble with the design. Because I did the design. I know that it’s made of thousands of little decisions, thousands of little trade-offs you have to make. I know that it’s a hard problem and that it’s easy to make mistakes or to fundamentally misunderstand something about the mental model the users have. So any information about what works and what doesn’t is extremely valuable.
your company is the exception. Most companies do no user testing. The closest i came was working at a co that did user interviews and ran some mock-ups by them before turning to eng. Once it was in our hands there was no change based on user feedback, because there was no user feedback before release. Not because we weren't willing to, but because ... they just never did.
Most folks i've spoken to do no user testing whatsoever at their companies.
I haven't seen much evidence that would say otherwise, but there's plenty of circumstantial evidence favoring my belief. Like this case.
98 and XP were basically just flavors of win95 and those were everyone's favorite, UI wise.
I still think Windows 3.1 was peak UI though ;)
Then, after launch we'll do rounds of optimisation based on the data we get back. This bit doesn't happen as much as we'd like as newer projects take priority, but it does happen and we're working on ensuring the Build > Measure > Learn loop is an integral part of the process.
We're lucky as senior management understands the benefit of testing and iteration. One place I used to work, the chief exec thought they knew everything, so didn't understand why we wanted to research and test stuff all the time. So we ended up having to do what they wanted, rather than what we knew the customer needed.
Too, too true. I have seen designers proceed to "educate" their test users (sometimes for weeks at a time) as to why the users' preferences are wrong and the designer's are right.
1. Come up with a design they think will work 2. Do user testing looking for confirmation that it works. 3. Change things (based on user feedback) in their design until they find confirmation.
I've come across this, and I wouldn't say it's data driven. It's, if anything, data supported, but it is still sounds like the initial design/idea still comes from within and there's a chance that it's down to their taste, interests and convictions. Or worse, a trend. We tend to pitch our ideas along with the data to support them so it's natural to seek confirmation as part of our normal workflows, but that's very different to taking design decisions based on telemetry, user feedback, etc.
Now, it's possible that I'm completely wrong and your UX teams actually does that. For instance:
1. You get support tickets from users about a missing or difficult to use feature. 2. This prompts team to design act. Design decisions are made, user testing happens, feature or changes are released. Once released, you notice from your telemetry data that the feature is rarely used by the wider audience, perhaps some of its UI items are used more than others. 3. UX team goes back to the drawing board to try to improve visibility, and does user testing.
Success rate is usually higher in that situation since you know users actually want that feature or change, and it's just about getting it right.
In Slack's case, I'm left wondering if any users actually wanted this change.
They never backed up their claim with any data, and instead released a very partial plugin to shut up dissent.
Basically all they do is look at Adobe UIs and copy them, even if IDEs and Photoshop are not used by the same people and not for the same usage ...
We did the whole bit: Hired a special company that had rooms with one-way mirrors so you could watch the users use the thing.
It was brutal.
One person went to this really cool subpage, and we were all so excited to see what they were going to do. User waved the mouse around for a few seconds and then clicked back. We were jumping up and down in our seats, just losing it. Our special thing was so cool (we thought) and this person literally couldn't begin to understand what it even was.
Whether it's useful really comes down to who is personally attached to the design in question and how much institutional power they have.
I've seen games trashed in testing only to have the lead designer double down on their designs; which always goes about as well as you'd expect.
It's amazing to see a game design fundamentally fall apart when users get a hold of the product and then see the game designers tearing their hair out. They do usually double down and call the gamers dumb or something lol.
If we didn't do this we'd lose to the competition.
When I worked at EA we regularly performed user validation, we had a whole room set up for proctored user testing. Hired professionals to conduct the testing. We also invested a heap into A/B testing and had a whole team dedicated to tracking this and analytics in general. This was just for marketing/launch web sites.
I've been working in the industry for 20 years and while the norm is as you describe there are certainly plenty of exceptions, especially when you product lives and dies by UX.
That doesn't mean everything necessary gets chucked out immediately. Sometimes we'll then test individual parts of a design (failed or otherwise) to see if those do better on their own. And complete redesigns with similar aims do sometimes get created (but they're usually completely different in colour scheme, layout, text, etc).
We also do user testing for the same stuff. That too is more important than a designer's opinion would be.
So yeah, it does happen.
Funny thing too: because I usually do the implementation of my own designs (react-native app so not so big of a barrier), I usually discover limitations of the original design and have to tweak it to better fit the medium.
My teammates just came back from an on-site user test, and the lead designer's own report describes how some features which he had pushed for were not working out for the users.
We have always done user tests, and we have always applied the results if they were meaningful, from balance to UI tuning to, sometimes, scraping entire features. There's plenty of companies that do this, and individual designers that really care about the user more than their own ideas.
lol.. I don’t know how, but this wool has been pulled over everyones’ eyes. These “data driven” decisions are often anything but. The people making them usually don’t even have sufficient background in stats to be capable of making them. They just plug and chug in some NHST framework or A/B testing framework, making all kinds of test errors or even outright cheating since their job incentives, much like failed research incentives, requires a constant stream of new positive results that causally drive growth & engagement. Since they “have to” find these things, then anything which can be politically argued into the product will be made to “have the data to back it up” (even if it doesn’t really).
The bigger the company, the worse this effect gets.
Do you remember Skype? Everyone used to be on Skype. Who uses Skype anymore today?
I get little pangs of dread a split second before I realize I'm about to have to spend 10-30 seconds figuring out new formatting since it shipped.
And as a response, I'm less reluctant to use it, meaning my messages are less detailed, meaning longer conversation, less precision.
It's a "convenience" for non-power users, on a feature only used by powerusers
Right now, they just basically blew away something that worked really well for alot of people and are demanding their captive audience adjust.
Whoever at Slack pushed this and threads has a good case of the Jony Ives going on. Thats two UI/UX changes that have met with lukewarm reactions to absolute dislike.
Give me a simple text edit option (heck, even Jira lets you choose between the two because they understand users have different preferences) and I'll be fine.
This video[1] from slack.com front page has 3 chat windows. 1 of them is about two engineers chatting about a git pull request, so I guess 1/3 of their target persona is a developer?
[1] - https://a.slack-edge.com/085e3/marketing/img/channels/vid/ch...
Of course, it's not unlikely that they weren't so much afraid as that they thought nothing would be done with it anyway.
You have to remember Slack's target persona is probably no longer the Engineer (If it ever was) - it's more likely a much less tech-savvy employee who finds WYSIWYG editors very handy to create rich text inputs.
I guess my point is - I'd wager this wasn't "rail roaded" through by some senior stakeholder that no-one can speak up too, but was probably a decision made by a product team who have the data to back up their decisions.
Now if the above isn't true (and perhaps the opposite is true) - then agreed, those are the signs it's time to leave.