Re:30% of Google's Emotions Dataset Is Mislabeled
by codedokode on Thursday July 14, 2022 @05:10AM
(#32093731)
You can filter out bad workers by preparing an additional well-labeled dataset and removing those who made even a single mistake in it. Also you can give the same task to several workers and check if they give the same label. However this won't protect against a bot using multiple accounts and giving answers based on a hash of a question so that the same question gets the same answer in every account.
Re:30% of Google's Emotions Dataset Is Mislabeled
by stjohnswarts on Thursday July 14, 2022 @10:55AM
(#32096822)
"removing those who made even a single mistake" means you don't want humans working on this lol. Humans will always make mistakes.
This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying more could overcome that because so many people engage with the platform in bad faith.