Re:30% of Google's Emotions Dataset Is Mislabeled
by MonkeyMalarky on Thursday July 14, 2022 @09:20AM
(#32095740)
I'm eager to see what nefarious things Google will do once they've mastered the art of identifying all the busses in a photo!
Re:30% of Google's Emotions Dataset Is Mislabeled
by a9h74j on Friday July 15, 2022 @02:24AM
(#32105215)
What tells me to start worrying if they get very good at identifying paperclips?
Re:30% of Google's Emotions Dataset Is Mislabeled
by stjohnswarts on Thursday July 14, 2022 @10:49AM
(#32096765)
their motorcycle, hills, signal light, and traffic light AI will break the world when it's over!
Re:30% of Google's Emotions Dataset Is Mislabeled
by htrp on Thursday July 14, 2022 @04:49PM
(#32101275)
> To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300.
are you me?
I did the same thing (worked as a mTurk labeler for 2 weeks) which convinced me to never use mTurk for anything even remotely important.
I've been able to use semi-supervised approaches with actual domain experts reviewing outputs.
Re:30% of Google's Emotions Dataset Is Mislabeled
by drewcoo on Thursday July 14, 2022 @10:18AM
(#32096381)
There are ways to detect problems with results due to things like fatigue or attenuation. And there are ways to deal with those problems (like forced break time and switching up tasks).
It's a wonder that none of that is built in. But maybe things like MTurk aren't built to maximize worker effectiveness because it costs too much. Are there better quality crowd-sourcing options?
Re:30% of Google's Emotions Dataset Is Mislabeled
by simiones on Thursday July 14, 2022 @10:54AM
(#32096820)
> Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future.
I for one always try at least once to get something wrong, often several times if it doesn't go through immediately, depending on urgency of my task. I hate being made to work for someone else like this.
To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300.
It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc.
All jobs were below 1st world minimum wage and were incredibly depressing. I think the adult content ones were the worst.
"Jian Yang: No! That's very boring work"
Worst thing was that you could not stop to think, you had to keep going if you wanted to at least make a few bucks an hour.
There are two solutions to the labeling issue: Pay well (at least $20 an hour no matter the location).
If the workers feel exploited you do not get good results no matter the sanctions.
Even better solution to labeling is to find people who give a shit about your task.
That is how I've done 50k lines of labeling training our OCR for a rare font for Tesseract. We have a few volunteers and they know this is important work (preserving cultural heritage, non-profit, national library, etc).
Old reCaptcha had a similar "feel good" element.
Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future.