There should be ratings system in Mechanical Turk based on sincerity of answers. You should be able to pay more to get people with better ratings.

Another option is to get reviewed by thousands of people through Hackernews, Reddit, Facebook, Twitter, etc.

Feedback from person 201 for free:

I love the idea but I'm really hesitant to sign up for such broad categories. What would get me to sign up would be allowing me to enter keywords like:

Physics, Austrian economics, marketing, Ruby, Android, Spanish fiction etc.

I'd be fine with those even if the newsletter only had one book in it or if it came only once every couple months as new books were released. As it is, getting a newsletter with all fiction or programming or science or business books in it wouldn't be much better than sending spam with random Amazon links.

Great idea though and the site looks really nice.

Thanks for your feedback. I think that the service as it is works for some people, and it doesn't for others.

That's why my next big feature is going to be adding the ability to follow specific authors, and probably keywords as well. This way people will have a choice. :)

Just FYI I signed up for your book list, what to read next is an issue I have a lot, and with a life like mine getting time to hit a bookstore is a luxury I rarely have.

Also, Go Canada, nice to see another startup joining the ranks from my country. Good luck to you!

The motivation to visit your website is 17/200 $, so this is a crucial information. You can't use this sample for inference about the real population of readers or customers.

A better approach for those 200 people would be to ask them: i) who do you think would be interested in this site and why. ii) What other services are related to this one. iii) Can you tell a source of clients for this site?

Anyway, a crooked sample doesn't allow a straight thinking or inference. The number of people 200 is not related to the 17$.

As I mention in the blog post, this is not a statistically valid sample. They can, and did, provide me with food for thought nevertheless. That said, your questions are all good suggestions for follow up tests (keep in mind that you need to phrase them in a A/B format).

I just wanted to offer a suggestion regarding the colour scheme - I personally found the blue menu bar on a blue background (with a blue search box) to be quite hard on the eyes; I think it would be worth spending a little time experimenting with different colours there. Maybe invert the menu bar, so it's mainly white with black text and a blue highlight for the current page? You could probably leave the search box as it is in that case.

Anyway, best of luck with the site :-)

Is there a problem here in listening to the wrong people? That author admits that this demographic is quite different from what might emerge organically; might their answers be equally as different? I'm not downplaying the inventiveness of this method for getting feedback, but I'm worried that it's a little misguided to use this for development purposes.

Glad you had a good experience with the service! It feels good to know that we're helping other entrepreneurs out. :)