seems like the situation is ripe for such a tool
---------
start with a subset of questions you have a predetermined answer to, only keep feed questions if the person responding has met a certain quality threshold on those question
every so often feed them a QA question
every so often send the same question to someone else to check it for redundancy
seems like there is a lot you could do to adjust theses based on Bayesian confidence intervals, and exactly how mission critical you need certain data to be
maybe something like that already exists, idk
------------- (edit: is this what the Scale API does?)
All that said, Scale API seems like a nice alternative to Mechanical Turk for some kinds of tasks.
The pricing page isn't totally helpful. I'm not sure if I would fall under `Comparison $0.10` or `Data Collection $0.25/minute`... or if many of those prices apply to a single job.
Either way, I'll keep your product in mind.
It was a bad enough experience that I'd never base my mission critical product on a 3rd party startup. Amazon could go bad too but a small startup it's almost expected.
I'm not sure I would do it again though. A lot of the businesses we were targeting don't have a web presence and therefore "No URL/No Email" was a viable answer. However, when I went through the list to see 150 "No URL/No Email" answers I didn't know for sure whether that is true or whether the Turker realized they could just copy/paste and make a quick buck. Amazon does provide the amount of time they spent on the task so I rejected any that were less than 10 seconds as I felt like they didn't give it a good enough try. Over that, I just accepted the answer realize that it may be false.
In the end I think I spent more time going through results and correcting them then it actually saved me. I'm excited to use MTurk in the future again, but only for appropriate projects.