News for hackers. Stuff that matters.
Re:Your Startup’s Office Is Missing a Room by potatolicious on Friday July 19, 2013 @04:09PM (#6072635)
I agree. This sort of usability testing is valuable - more valuable than simpy A/B testing in a vacuum - but it's important to remember that this is extremely qualitative and unscientific.

Throwing in the webcams, eye-tracking, or what-haves-you IMO incorrect lends an air of specificity and scientific-ness that isn't warranted.

Before we launched our last product update we brought people in to use the app. It was very much as you described - we invited them to come into the office where our PM sat with them, watch them used the app, and asked good questions.

The much more subtle behaviors are IMO not determinable via pseudoscientific things like watching facial expressions, eye movement, heart-rate, or whatever these things are supposed to measure. Best A/B test for the rest.

Re:Your Startup’s Office Is Missing a Room by soneca on Friday July 19, 2013 @04:15PM (#6072673)
I totally agree. This is a qualitative experiment. You try to generate insights, assumptions, hypothesis from it. Then after that you test these insights more scientifically.

Including this, very well put, pseudoscientific things, on the qualitative phase must do more harm than good. Maybe they act as an excuse not to really test the assumptions.