This was the first Cambrian House business model.

I saw two problems. First, you didn't know what to do with your metrics -- if changing a button from green to blue caused signups to go from 20/day to 25/day, which number indicates how much customer demand exists? Second, as pg pointed out, startups rarely get very far before figuring out there's actually a much better product for them to build that's about 70% different from their current one. At best, the site no longer is evidence of interest in the new product, you have to start it over. If you've taken pre-orders (the only way to reliably gauge interest) you may be committed to building what you sold.

There's another important step:

Drive customers to the site AND make sure they are the real market your targeting before taking their feedback.

Good points. I've "expressed interest" in a bazillion things, but I'm loath to part with money. I sign up for information announcements and beta sites because I'm simply curious about stuff, but I may not be a good customer when you get to the bottom line.

This is an old trick out of the direct marketing handbook, BTW. Direct marketers would send out mailers for lots of different products, asking people to respond if they were interested.

They then gauged market demand by the response rate.

I think this technique might actually be illegal when done over snail mail nowadays.

In B2B sales there's also a significant gap between "interest" and "purchase". You have to be able to tell how much one implies the other.

I think that in general, these sort of tests need to be seen as early probes, not absolute measures. They might be better in a relative test:

"Should I build this or this?" rather then "Should I build this?"