It's certainly been my observation that cynical developers who test things as they go by deliberately putting in silly things into stuff they just wrote seem to get hung up less in testing.
I mean the system I inherited at work the first thing I did when I got an instance spun up was put in a negative value in the quote line quantity (which immediately broke..well almost everything) then decimal values in quantity fields where only integers made sense, then text in number fields and so on each time breaking something in a new and interesting way.
Sometimes I think it's hard not to be cynical about enterprise systems.
My old lecturer (somewhat pithily) "Almost all the testing in the world means nothing compared to 15 minutes in the hands of the 17 year old office junior"
There's whole classes of highly paid engineers whose job is to do this. But they work for old fashioned, boring, companies.
Another part is hiring a breed of test engineers who like breaking stuff and have a knack for it.
It's one of the reasons why I don't trust myself to test things fully, we write the software with all sorts of assumptions in our heads and subconsciously steer away from doing silly things - in that context it's really difficult to aim at a point a zero-knowledge user would hit.
Moreso than any of the security tests I've written, that fuzzing has broken every enterprise API our customers have thrown at it.
On one of my teams, we had a hard rule where all features must be tested by 2 other team members (and if multiple people worked on a feature, none of them can be the testers). Something like every other case found something questionable, if not an outright broken edge case, that the developer(s) completely missed.
I thought this was so dumb it was brilliant QA.
He's the sort to dig as far into the internals of things as he can and then start messing with it (he implemented partial function application for C, for example [1]).
My poor prototypes never stood a chance.
It turned out that some of them, indeed, simply didn't care -- and didn't know, either. We'd explain what the problem was and they'd shrug or say they'd seen <big name software> breaking like that, too, you fix it if it turns out someone actually breaks it.
Others, however, would skip it so that they could focus on stuff that was more complicated or more relevant. They'd validate one set of inputs, just to show that they know it needs to be done and can do it, but not everything. Or they'd throw in a comment like //TODO: Validate this by <insert validation methods here>. Most of the time we'd just ask them to talk us through some of the validations -- and most of the time they could write them on the spot.
You could argue that this is very relevant in real life, and that even if it weren't, what's relevant is the interviewer's choice, not the candidate's (although tbh the latter is one of the reasons why tech interviews suck so much).
But at the end of the day it is an interview setting, not a real-life setting, no matter how much you try to make it seem otherwise. At the end of the day, the people doing it are still young candidates trying to impress their interviewers, not company employees working on a company project under the supervision of tech leads. You don't end up with much useful data unless you allow for some flexibility in this sort of experiment.
Your 1st point isn't correct, in that you will learn interesting things from 1m users that you won't from 15. The thing is that the 15 will tell you why, whereas the 1m won't unless you ask precisely the right question (which is a problem in itself). It basically takes experience here... (And this is something an AI may eventually become very good at.)
Your 3rd and 4th point aren't correct, in that given some sampling error you may very well find that what 15 people don't like 1m will.
I'm in full agreement with you in principle, in the sense that I firmly hold that far too many start-ups do quantitative without qualitative and end up with the wrong conclusions; but you can't just wave away quantitative data like that.
It's a nondescript icon made of three horizontal bars, it looks like literally anything that comes in threes. My mom calls it the pancake button and my fourteen year-old nephew used to call it the button with something that looks like a fork but without a handle until he switched to the "meh" button once he became a nihilist (teenagers do that stuff sometimes). A menu with three items is very likely to be among the last things that crosses a non-techie's mind.
At this point it's been used enough that anyone with enough exposure to electronics knows what it does, but it's hardly a better choice than the "File" menu. The point of making something intuitive, as opposed to explicit (i.e. by using a symbol as opposed to spelling out) is kindda missed if you need to "well actually" it and explain why it means whatever it means.
— History of Empire and Colonialism
— History of Religion and Society
— History of Race, Gender, and Power
⋮
— History of War and SocietySo it naturally means "there is more stuff here" to me. YMMV
Then again, the magnifykng glass icon for search didn't make sense either until I read the docs but at least back then people made docs and kept the UI stable enough so that it made sense to learn it.
I still remember fondly being good with OLE (Object Linking and Embedding) in a Microsoft Works.
I think we would be better off if we kept the best from both: the predictability and discoverability of the past with the niceness of today.
Why should we have to choose between nice and usable?
Indeed, on teams and products of any size there is always a decision to be made about which accessibility requirements will be met and to what degree.
I would make the separate argument that what you need with accessibility requirements is separate testing expirements for each one. You don't need 100 testers to to make sure you have some special accessibility requirements, e.g. you need 15 normal vision testers and 15 impared vision testers.
quantity has a quality of its own
Basic UI/UX will be significantly revealed be small-group testing.
Don't let the perfect be the enemy of the good.
That’s a big distinction, as when you look at a million people, 80% are are either the same, or don’t really understand what they are doing at all.
Modern designers usually assume that people are dolts and need simplicity. Optimal in my opinion is providing quality on-boarding and understanding behavior of someone who knows what they are doing as well.
My formula is therefore: Pick a few principles to apply to the solution you're making, make sure as best you can that they agree with each other. Test to see if potential users agree with those principles. Select features based on the resulting design framework. Then develop the product as a way of discovering more about this framework.
Lots of companies are doing this implicitly through their founding teams and hiring practices: they just end up with culture that values certain principles, so they get upheld at every meeting and the results end up in the product. But it can also be communicated and imposed more explicitly, and I think that's where design becomes more visible in the process.
If micro fails, macro will most likely fail.
At the macro level you’re tracking higher level metrics like how many users engage with the different parts of the UI, where the fallout is? What is the retention of users, what do power users do differently to new users e.t.c
This kind of wholistic view about the product at micro and macro levels gives you a much better understanding about the bottle necks in the product.
At 15 users your training budget is to teach everybody how to use your mess of a UI is cheaper than the cost of fixing the mess of a UI. At 1,000,000 users your cost of training is higher, but you also have more users to spread the cost of preparing the training among.
Maybe for many purposes, 5 won't cut it and you need 100 or 1,000, but you don't need 1%.
- a product designer/manager of something with 1,000,000 users won’t learn more about usability than a product designer/manager of something with 15 users. All those measurements of flows and secret at scale analytics data is sort of worthless for the purposes of usability.
- people with 15 users worth of learning about usability instead of 0 users are way undervalued, while people with 1,000,000 are way overvalued
- “we don’t know until we test it,” the #1 refrain of big company design nowadays, is intellectually bankrupt for most free software, since if it looked bad for 15 users it’s probably going to still be bad for 1,000,000
- after you’ve shown something to 15 people and they don’t like it due to usability problems, you’re extremely unlikely to find 1,000,000 more who will like it.
This also appeals to my instinct that there is something learnable about design, that great design is achieved by people and not by massive datasets.