Re:Evaluating Bing with Mechanical Turk
by shalmanese on Wednesday June 10, 2009 @11:10AM
(#651100)
With only 6 to 8 participants, the study definitely didn't have the power to find all but the grossest differences between query types. It's not about study design so much as the need to recruit more people.
Maybe a two-phase Turking might be interesting, too: first have the turkers fill come up with a few parameters for each query (e.g. current events, general info, celebrities, pr0n), then compare search engine results for those queries. That would help pick out the more subtle patterns that you were looking for in the original graph.