News for hackers. Stuff that matters.
Re:Mechanical Turk Lessons Learned by fenwick67 on Wednesday February 01, 2017 @03:14PM (#13544628)
I think you probably needed to have some redundancy, have each business checked at least twice.
Re:Mechanical Turk Lessons Learned by pitt1980 on Wednesday February 01, 2017 @03:22PM (#13544703)
are there any tools that help you do QA on MT data entry in a Bayesian way?

seems like the situation is ripe for such a tool

---------

start with a subset of questions you have a predetermined answer to, only keep feed questions if the person responding has met a certain quality threshold on those question

every so often feed them a QA question

every so often send the same question to someone else to check it for redundancy

seems like there is a lot you could do to adjust theses based on Bayesian confidence intervals, and exactly how mission critical you need certain data to be

maybe something like that already exists, idk

------------- (edit: is this what the Scale API does?)

Re:Mechanical Turk Lessons Learned by doleson on Wednesday February 01, 2017 @10:41PM (#13547913)
Yup. My old company (www.crowdflower.com) has a platform set up to do this.
Re:Mechanical Turk Lessons Learned by brilliantcode on Wednesday February 01, 2017 @05:57PM (#13546246)
Please continue, this is a highly valuable comment.
Re:Mechanical Turk Lessons Learned by giarc on Wednesday February 01, 2017 @03:40PM (#13544886)
You are definitely correct. This was just our first kick at the can. It was only $30 or so to test out the service. We learned a bit of info that we'll apply to future jobs (if we do it again). Such as, telling the Turker what to do if they don't find a website/email for the business.