News for hackers. Stuff that matters.

A comment on Everybody is spamming everybody else on Mechanical Turk. Parent comment

Re:Everybody is spamming everybody else on Mechanical Turk by yummyfajitas on Wednesday March 09, 2011 @07:15AM (#2304640)
What you typically do is include gold data in your dataset. I.e., define a HIT to be 10 sequential tasks, 3-4 of which you know the correct answer to. If a turker can't get the gold data right, their work is rejected.
[ Parent | on HN ]
Re:Everybody is spamming everybody else on Mechanical Turk by _delirium on Wednesday March 09, 2011 @07:38AM (#2304674)
An even better (and common) strategy is to reject the work that fails the gold-data test and to reject most of the rest of the work too, claiming it failed the gold-data test.
[ Parent | on HN ]