News for hackers. Stuff that matters.
Re:Case Study: Accuracy & Fraud on MTurk by sheraz on Monday February 06, 2012 @02:13PM (#3558740)
Interesting story -- I've attempted to use mturk a few times in the past, but it is actually a lot of work to get it right. Good instructions are really important, as well as a way to measure performance.

Good luck to them as I think they might have made mturk easier to use for people like myself who see the value but couldn't figure out how to "make it work."

Re:Case Study: Accuracy & Fraud on MTurk by MediaSquirrel on Monday February 06, 2012 @02:24PM (#3558781)
good instructions are crucial.
Re:Case Study: Accuracy & Fraud on MTurk by chrisyeh on Monday February 06, 2012 @06:53PM (#3559791)
I just used Humanoid for the first time. I had a job that would have taken about 5-10 hours (look up 50 ad agencies, record their Twitter account, blog URL, and their CEO's Twitter account and blog URL).

Humanoid accomplished the job for under $35. Considering I would have had to do it myself, or nag an intern for a week to do it, I consider it money well spent.

P.S. I don't know what results I would have gotten from going to Mturk directly, but the results from Humanoid were better/cheaper than my old habit of outsourcing to an Indian body shop.

(Disclosure: I am an investor. But I'm giving honest feedback.)

Re:Case Study: Accuracy & Fraud on MTurk by bitstormer on Monday February 06, 2012 @02:29PM (#3558804)
Great data, very cool how sophisticated algorithms can increase performance dramatically for human-powere tasks
Re:Case Study: Accuracy & Fraud on MTurk by garethsprice on Monday February 06, 2012 @03:42PM (#3559097)
What does the author mean by fraud? Do people submit junk results and hope they slip through the screening process, or it it something more insidious?
Re:Case Study: Accuracy & Fraud on MTurk by kailashbadu on Wednesday February 08, 2012 @06:31AM (#3566143)
Junk results, mostly, yes. That way they can complete more taks faster and still make money because most of the time their work is not manually approved.
Re:Case Study: Accuracy & Fraud on MTurk by MediaSquirrel on Monday February 06, 2012 @01:55PM (#3558664)
"Humanoid’s software identified 8% of the workforce as engaged in fraud." <--that is a big number!
Re:Case Study: Accuracy & Fraud on MTurk by gcb on Monday February 06, 2012 @02:41PM (#3558851)
Fails to mention how much they cost on top of the mturk prices to the client