In the past day or two, I discovered that a large number of responses in my latest Mturk survey appear to be random responses. I detect that a large portion of these random responses have repeating GPS locations. So far, a relatively large number of social psychological researchers also seem to have noticed a drop of data quality in their Mturk data and detected concerning patterns using the GPS method (see the related discussion threads on Facebook PsychMAP).
Re:A Bot Panic Hits Amazon Mechanical Turk
by herogreen on Saturday August 18, 2018 @10:10AM
(#17788527)
I find this article interesting because I did not know that MTurk was used by psychologist to survey people. The only usage I had heard of was data labeling for machine learning. This surveying usage seems very strange because I expect the data to be very biased: the (human) respondents will be located in places where the average salary is close to what is paid by MTurk, will be abe to use a computer, etc...
Re:A Bot Panic Hits Amazon Mechanical Turk
by sireat on Sunday August 19, 2018 @05:16AM
(#17793424)
One thing that the paper does not address is MTurkers becoming professional survey takers who recognize typical psychology surveys thus skewing the responses.
Personal anecdata: I took many MTurk surveys for about a month in 2012. It is mind crushing work.
There were countless near identical surveys presumably by psychology undergrads.
Pretty much anything in Kahneman's Thinking Fast and Slow was there.
Very quickly you recognize the typical priming experiment and then is your response any valid?
If you know the idea behind the baseball bat and ball pricing question then is your correct answer made in a split second helping?
Most survey givers wanted MTurkers with high ratings.
If you think about it, those MTurkers with high rating would have likely seen all those survey questions already.
Thus experienced MTurkers would really distort the answers compared to a random sample of undergraduates.
In the past day or two, I discovered that a large number of responses in my latest Mturk survey appear to be random responses. I detect that a large portion of these random responses have repeating GPS locations. So far, a relatively large number of social psychological researchers also seem to have noticed a drop of data quality in their Mturk data and detected concerning patterns using the GPS method (see the related discussion threads on Facebook PsychMAP).
http://journal.sjdm.org/10/10630a/jdm10630a.html
Effectively, without the bots, the paper contends that the Turkers are at least as representative a sample as undergraduate students.
Personal anecdata: I took many MTurk surveys for about a month in 2012. It is mind crushing work.
There were countless near identical surveys presumably by psychology undergrads.
Pretty much anything in Kahneman's Thinking Fast and Slow was there.
Very quickly you recognize the typical priming experiment and then is your response any valid?
If you know the idea behind the baseball bat and ball pricing question then is your correct answer made in a split second helping?
Most survey givers wanted MTurkers with high ratings.
If you think about it, those MTurkers with high rating would have likely seen all those survey questions already.
Thus experienced MTurkers would really distort the answers compared to a random sample of undergraduates.
Maybe you can compensate for that somehow.