I started reading this as "wow, they live in a very different world in SV where you can advise startups to set up a room like this". I am in Brazil and here there are lots of startups with founders working at home, not even an office, much less a usability room. At the end, at least the author sugests that this is for post-series A.

But then I remembered, I did this myself! At home. With my father. In a much leaner way.

I am a sole founder, working at home, while my wife pay for our bills (all that to say, I have no money to invest). My product is sort of "online pills dispenser as a service", a SaaS B2C helping elderly people organize their medicine. And my father is just my target (actually, I started all this thinking of solving his problem). When created the first version I went to his home and let him use the site. I just typed the URL and then just kept watching. He pains attention to details, so I left, one hour and a half later, with literally more than 50 notes of changes that I should made. From small design and copywrite details, to whole new features.

And you don't need webcam, microphones and a soundproof glass. This is the case if you have bosses for whom you must report. If you are the founder, just you, there, taking notes(and asking the right questions) while the user experiment your product, it is enough.

If your friends are the target, your house is fine. If you must talk to strangers, a public computer is enough. Even Eric Ries on his book talks about customers going to their office to do this - not a whole "usability room". Sometimes I think funded startups spend their investor's money just for the sake of it.

I agree. This sort of usability testing is valuable - more valuable than simpy A/B testing in a vacuum - but it's important to remember that this is extremely qualitative and unscientific.

Throwing in the webcams, eye-tracking, or what-haves-you IMO incorrect lends an air of specificity and scientific-ness that isn't warranted.

Before we launched our last product update we brought people in to use the app. It was very much as you described - we invited them to come into the office where our PM sat with them, watch them used the app, and asked good questions.

The much more subtle behaviors are IMO not determinable via pseudoscientific things like watching facial expressions, eye movement, heart-rate, or whatever these things are supposed to measure. Best A/B test for the rest.

I totally agree. This is a qualitative experiment. You try to generate insights, assumptions, hypothesis from it. Then after that you test these insights more scientifically.

Including this, very well put, pseudoscientific things, on the qualitative phase must do more harm than good. Maybe they act as an excuse not to really test the assumptions.

I think it has to do with culture: They don't spend it exactly for the sake of it but because rooms with shiny iMacs that get featured on blogposts like this one are usually the validation of one's investment.

One trend I have noticed a lot in Berlin is money being spend in all sorts of things that are in no way related to the product or service the startup was trying to create - and the few times I dared asked I got the same "they are investing a lot of money on us man, that party should better show it!".

I think you are right. The founder thinks: "If I don't spend all their money, they will think I raised more money that I needed, which is probably true, but they don't have to know."

While the investors think: "If they don't spend all our money I will think they are lowing the chances that this investment will pay back, so they better spend it all".

You can get a ton of value out of usability tests without going all-in on a usability lab. Buy screenflow and a USB microphone to record sessions, and then recruit people for $50 a pop to come in and go through your product.

Steve Krug calls it "Lost Our Lease Usability Testing" and explains how to do it in Don't Make Me Think. You can download the relevant chapters here: http://www.sensible.com/downloads-dmmt.html

We did this at a company I worked at a few years ago and it made us completely change how we were building the product because our target users got nothing out of it (I talked a bit about this a RubyFringe: http://www.infoq.com/presentations/francl-testing-overrated)

The sequel to Don't Make Me Think is entirely about how to do quick usability testing. It's called Rocket Surgery Made Easy, and I highly recommend it. He walks through each of the tools used, how to structure questions, and all of the other important pieces.

It made starting my grad work in usability much less painful.

I don't think a usability lab is that expensive. A few webcams, a mike, plus a laptop and mobile phone is all you need. That probably totals to at most $3k.

Both kinds of feedback, lab and cafe, are useful. They each provide a different kind of feedback, from a focused and unfocused user so they aren't mutually exclusive. Rather, both are worthwhile.

My point is that the article calls for post-Series A companies to have "a soundproofed room with a table in the center flanked by two chairs, one for a user and one product manager or UX researcher" with "a constellation of web cams".

You don't need that to get value. Initial investment for "lost our lease" usability testing is $99 for Screenflow and $70 for a Snowball USB microphone. It would help to do the test in a room that's not full of people, but grab a conference room, or do it after work.

There's no reason a bootstrapped startup can't afford this -- in fact, there's an argument to be made that they can't afford NOT to do it.

Keeping an extra room costs way more than that. It doesn't make any sense for most companies to have a dedicated lab, you're not going to run usability tests all day, every day.

Plenty of market research companies offer rooms like this for rent.

It doesn't cost that much, really. The salary of the guy running the lab is the largest expense by far. At a startup I worked at a few years ago, they had a lab running 40 hours a week. It would have cost about 10 times as much to have a market research company do this for us if it was even possible.

A couple companies ago we did it the CEO's office. Over the course of a day we brought in a half dozen people recruited from Craigslist. That one day alone gave us a lot of useful info.

I wonder if there is a market for "usability screen recording" software, which records simultaneously from both the screen and a webcam, and lets you play back the two streams in a split-screen format.

Screenflow can do this. http://www.telestream.net/screenflow/

I thought about starting a company to do exactly this. The near term would be to offer it as an in-person service (consulting), then use that experience to develop an MVP.

There are a few companies doing this already, so not exactly new.

When I worked for Citrix Online I took part in internal usability studies. It was great to be able to sit with a project manager and explain what made sense to me as a user and what didn't. It was also thrilling to see a suggestion I made make it into the final product a year after I left the company!

Is it the perf room? Please tell me it's the perf room.

(Relevant anecdote: where I used to work had a makeshift UI rig with webcams and laptops. It could be dragged into any conference room for bringing in end-users (teenagers) to try out new games/experiences/interfaces in progress. How do yo find end users? Offer $5 to $20 gift cards on craigslist for their time if your product is broad enough.)

This comment made me laugh -> the perf room!

In my 15+ years experience, I've found 1-on-1 conversations at a shared table (sometimes even at a coffeeshop over coffee) to be provide just as actionable and informative feedback as those of a lab environment. Add in the dramatically lower price tag and the honesty of a human-to-human conversation and I head to the coffeeshop everytime. I do believe there are one or two scenarios where the lab is preferred. I try to steer away from those projects.

>the honesty of a human-to-human conversation

i think that the dishonesty of human-to-human conversation is exactly the problem that usability labs are trying to solve. People are way less likely to complain about your product to your face, they'll keep insisting it's good and you'll have to work harder to drag the problems out of them.

And yet, and yet... the new Gmail.

...passed through extensive usability testing. just because you don't like it, doesn't mean it sucks. that's pretty much the entire point of actually doing testing rather than relying on some guy's personal opinion.

Ah, but I, and my friends didn't like it.

QED, right? ;)

There's someone in Cincinnati who is doing this in a much, much different fashion -- they're buying drinks in return for 1-5 minutes of interaction with an iPhone app. It's a really unique way of getting instant feedback on his app. I've noticed that he's semi-responsive to his changes -- he'll be back a few days later with a new build, and a sbux gift card, looking for more input on his changes.

One of the problem we face now on our product is same thing and we also do a similar excercise to improve product usability. But what we found was most of the suggestions were purely subjective and biased. Picking up right audience is the key. For example, instead of calling a CEO who is going to buy my product to evaluate i would rather invite someone who will use this product for living.

Not to disparage the idea but this can be expensive to set up and time consuming to use. Also, it can take a while to get the process down and/or have someone who can properly interpret what the user is actually doing.

I have found that in early stages of a startup that one or more subject-matter experts who are hopefully involved with the space can give much more direct feedback in the early stages.