30% of Google's Emotions Dataset Is Mislabeled

Re:30% of Google's Emotions Dataset Is Mislabeled by tgv on Thursday July 14, 2022 @04:18AM (#32093412)

This kind of out-sourcing is quite common, and at the prices I heard some time ago, you could run each utterance by 3 to 5 people, which allows you to get an idea of the reliability.

But the core of the problem is 27 emotions. That's really asking for trouble.

[ Parent | on HN ]

Re:30% of Google's Emotions Dataset Is Mislabeled by RabbitUpRabbit on Thursday July 14, 2022 @05:00AM (#32093673)

> But the core of the problem is 27 emotions. That's really asking for trouble.

I'm not sure I agree. The examples highlighted in the blog post aren't cases of slight mislabels (like mislabeling frustration as anger, for example). They are often labeled polar opposite to what they should be.

Though perhaps what you are saying here is that low-paid workers won't bother to look through a list of 27 emotions to find the right one, and thus they are more likely to label at random.

[ Parent | on HN ]