News for hackers. Stuff that matters.
Re:30% of Google's Emotions Dataset Is Mislabeled by Thorrez on Thursday July 14, 2022 @03:14AM (#32092960)
>Farm it out to low-paid non-native speakers

The paper claims:

>“All raters are native English speakers from India.”

Re:30% of Google's Emotions Dataset Is Mislabeled by pmontra on Thursday July 14, 2022 @03:52AM (#32093235)
US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.
Re:30% of Google's Emotions Dataset Is Mislabeled by nmfisher on Thursday July 14, 2022 @04:01AM (#32093297)
The blog post highlights this specific point - "US English" and "Indian English" really aren't the same English (in fact, I'd probably go even further and state that "Reddit English" and "US English" probably aren't the same English either).

Likewise, the Common Voice English dataset isn't great for ASR training outside India, either. There's a huge proportion of Indian speakers, and their data doesn't really help train ASR systems for non-Indian accents.

Re:30% of Google's Emotions Dataset Is Mislabeled by david_draco on Thursday July 14, 2022 @04:20AM (#32093426)
Is the "right answer" a classification based on US English with background knowledge of US cultural background, or is the goal to build a global sentiment data set?

You and OP ("the indians labelers don't know how to do it correctly") seem to want the former, so it would be good to state that goal upfront.

Re:30% of Google's Emotions Dataset Is Mislabeled by zharknado on Thursday July 14, 2022 @10:16AM (#32096364)
I think it’s valid to question US-centrism broadly in datasets like this, but presumably that choice was made in selecting the Reddit sample, which is dominated by North American English.

A model is more useful if it approximates the speaker’s intent, not the listener’s interpretation. “Right” is complicated in language, but it’s hard to see how you’d use a model full of cross-cultural misunderstandings.

Re:30% of Google's Emotions Dataset Is Mislabeled by bryanrasmussen on Thursday July 14, 2022 @03:37AM (#32093125)
https://www.heritagexperiential.org/language-policy-in-india...

>English, due to its ‘lingua franca’ status, is an aspiration language for most Indians – for learning English is viewed as a ticket to economic prosperity and social status. Thus almost all private schools in India are English medium. Many public schools, due to political compulsions, have the state’s official languages as the primary school language. English is introduced as a second language from grade 5 onwards.

Re:30% of Google's Emotions Dataset Is Mislabeled by fluidcruft on Thursday July 14, 2022 @11:15AM (#32097079)
Perfectly logical choice if you're building a machine to replace call centers.