The alternative is design by an elite group with a strong opinion. I'm very much in favor of that model provided that this elite group is knowledgable about all the tasks that users actually want to perform.

I think that's possible for consumer electronics but not for general purpose computing. No one can design for data center sysadmins, data scientists, financial analysts, architects and mobile salespeople at the same time without making mistakes.

Being opinionated in an area where your opinion is irrelevant is just being stupid.

I don't understand how doing user testing is necessarily "design by committee". Regardless of how you come up with your ideas you need to test them on actual users in order to benchmark what works and what doesn't.

That sounds correct - but has it actually been proven that you should design around simple indicators from user testing?

The general usage patterns of iOS were never developed with user testing at the core. Same for Android Holo. These designs were created directly from a designers intuition and have since been improved slightly with user feedback and testing. If you read up on publicly available information about the process though, it is clear that user testing was not the central issue.

Meanwhile whenever you read anything regarding Windows 8 or Unity design, the use of user testing was always core to the design process - often toted as the most important part. Just read this article and the main theme of their design discussion is on user testing.

Basically, I think this area needs more research rather than an off the cuff "you need to test them on actual users in order to benchmark what works and what doesn't". Sounds correct, but has anybody actually tested that? Often your test users don't see the big picture and don't really know themselves what they want. They're just answering 'big blocks were easier to click on'. That doesn't mean you must use big blocks - you need to consider the interface and usage as a whole, not simple data points from testing and consider your design done...

I think it depends on what you mean by "user testing". Rather than focus groups and asking users to design to app for you (which would be bad) I assume what is being done would be to provide a few sample interfaces and record users doing various tasks. You can also track eye movements and see which parts of the interface they concentrate on most.

If they take 10 minutes to figure out one interface and 10 seconds to figure out another one for example it's quite obvious which one is more intuitive.

What do you understand user testing to be?

I understand it (someone please correct me if I'm wrong) as being the same as usability testing. In other words, putting users in front of a particular design with a task you want them to complete. You then observe anything they struggle with. You can also measure how long it takes them to complete the task. These things inform you about how good your design is at the same time as making some design flaws very evident. One thing that is sometimes included (but I personally disagree with) is asking the user for an opinion.

However, consider the following:

A UI for creating a shopping cart. We can rank two different variables here:

1) Number of users making a purchase. 2) Number of users returning goods because they didn't actually want it.

If we focus only on the first option in our user testing (as any shopping cart user testing will do), we will generally make the 'fine print' descriptions of items as small as possible. We will make it very easy to buy an item with a single click. This is going to give us the best 'purchase rate', guaranteed.

However, we are going to get increased returns as users will miss important product details. We will get lower 'user happiness' with our store as users will feel a bit cheated and have to turn to external sites for more information. In the long run, users may stop using our store entirely and our purchase rate will fall.

This is a classic example of 'short term' or 'greed' based optimization. I feel nearly all UI testing I've seen does not cater to variable B - only to variable A. Catering to variable B isn't even really possible with a UI or A/B test?

If one of your key metrics is rate of returns, surely this would show up in an A/B test. You could also check for it in usability testing by making the task something where the user would be interested in the "fine print details" if they were visible enough for the user to notice.

iOS and Android(apples)are both easier to use and nicer to look at than Windows 8 and Unity(oranges)

I would absolutely loathe to run my laptop, desktop, or server on a mobile OS.

Point conceded - was just going with current 'talked about' OSes - but you're right in that they probably can't be compared.

I'd still like to see research done using the following - results could be very interesting.

A designer is told to create a screen for some fairly complex feature - something that has at least 10 inputs into some kind of algorithm.

A

Designer is given free reign to just make up some UI entirely how he thinks it should work.

B

Another designer is told to make different versions which are then tested by users. Users should be asked what and how they want the UI to work and how the different versions should be changed. Based on popularity of opinions by testers, that design should be changed to match what most testers prefer. Should have at least 3 repetitions of user testing and feedback.

After this, A and B are given to a new unrelated testing group who need to rank the UI from both A and B without being told about either or having anything explained.

Common sense says B should be better, but I don't think it's guaranteed... Anybody in a university position currently in a position to set something like this up? The results should likely be worth a research paper regardless of outcome!