Are they 1:1 your peers, colleagues?
Or are they underpaid, most likely emerging economy countries, and very overworked labor that you're paying for?
You can browse to the mechanical turk subreddits and you'll hear the other side of the table that they spend 15-20-hours on a task to get what, a few dollars?
Pay more, get better results. The system itself is designed to exploit.
For doing research, I think Prolific [1] is a popular alternative.
They include features like pre-screening based on demographic features (including participants' approval rates in past studies) [2], and attention check questions [3].
[2]: https://researcher-help.prolific.co/hc/en-gb/articles/360009...
[3]: https://researcher-help.prolific.co/hc/en-gb/articles/360009...
We're happy to have a conversation - feel free to reach out to me via my profile email or you can find our contact details at https://www.tasq.ai/
I created Cogmint.com ("cognition minting") to solve this problem for myself.
You can submit known correct answers for questions, and those questions are then used as ground truth to score worker accuracy. Workers are then scored on their similarity to known correct answers and other workers that have accurately answered questions. It works surprisingly well for how simple it is. It's been a fun challenge to create simple methods of scoring similarity across different task types.
It's a side project, so don't rely on it for mission critical things, but I rely on it for some production tasks, so it's stable.
It currently supports classification (choose from a set of possible answers) and has beta support for bounding box task types. String input task types are coming very soon.
I'd love to see if it can help you out, I'll waive the fees: I'm not in it for the money I just like making things useful and reliable. Reach out and say hi!
My undergrad thesis was to build https://tagbull.com, where we tried to have turkers validate the work of other turkers by breaking up a label into sub tasks, and getting multi-turker consensus on those before moving forward.
The main issue we ran into is that the incentive system is incredibly misaligned with the responsibility that the turkers have. It’s very difficult to build trust, especially with a crowd of people who haven’t signed contracts, and who face virtually no repercussions for doing bad work, whether intentionally or unintentionally.
Have you noticed problems that show up with questions whose answers have a bimodal distribution (ie. The gold standard question actually has two or more correct answers)?
In one sense, this is just a labeling quality problem with the 'gold standard' data, but to a lesser extent these same issues may crop up in the data being labeled when using similarity or clustering to rate or classify the workers and transitively apply that to the other results they produce.
I haven't dug into the data on this across the platform but you've given me the idea to go see if I can find evidence of this, and see if I can improve somehow. There's only low hundreds of projects, so I might be able to find some that have this problem.
Quality/metrics layer as a Saas with MT on backend, add a few percent to the price per work unit?
I know Stanford's research teams all use a common interface to mturk that keeps profiles of turkers on their side so they know who to solicit for upcoming surveys, conduct longitudinal studies, etc. I've always wondered why more universities didn't follow suit.
I built a side project called cogmint based on the insight that simple scoring and ranking of workers was valuable. I ended up building my own worker interface instead of using mturk because it wasn't much additional effort on top of the scoring logic I was building anyway. Perhaps other serious companies came to the same conclusion I did with my hobby project.
CrowdFlower maintained statistics about the accuracy of individual workers. Additionally, they made it easy to include gold-standard "test" questions, which weeds out workers who are not doing the specific task correctly.
Without this sort of quality-control platform, mechanical turk is just unreliable crowdsourced-labor infrastructure. I understand that AWS doesn't want to build too much on the services they provide but, really, mechanical turk sucks.
I'm working with a researcher who is conducting experiments using turk, and he has a half-written shoddy version of what crowdflower offered. It seems that everyone using turk has to reinvent the wheel. Why can't someone provide decent quality control over turk results?