I expect it's mostly a reflection on the learning data in language processing. Almost any corpus of text you can easily find for experimenting is close to standard English. Dialects very rarely appear in published texts unless it's just to point out that a specific person being quoted uses that dialect.

Choose a student of any colour. When they need a text corpus for research, do you think they'll reach for a different, known, large text collection that matches their background?

There may be some of that, but I think it has more to do with AI still being young, and it's easier to work with less ambiguous text for learning. That and everyday speech move much faster than formal writing. Now that my son has gone off to college and is less accessible as a resource, I expect to almost totally lose touch with the speech of "these kids today."

> I expect to almost totally lose touch with the speech of "these kids today."

Is this long term really a problem though? Regression to the mean: No one cares about the specific slang of the 60s anymore. And the hip slang of today will be out of fashion in 15 years.

In a similar vain the extreme T9 keyboard text message abbreviations of the youth in the 2000s did go out of fashion because of automatic spell checking and speech recognition of modern smartphones.

> Is this long term really a problem though?

How long term? What's "a problem"? We know with certainty that given the passage of enough time, the speech of "these kids today" will be completely unintelligible to you.

Do you think written Standard English is less ambiguous than any other English, or written language? Because I don't see why that would be the case.

>Do you think written Standard English is less ambiguous than any other English

Definitely. Things are clearer when you write formally and strictly adhere to grammatical rules. When I'm speaking casually I omit words and (ab)use punctuation much differently, and while that is perfectly acceptable to do, it is more complex and ambiguous to parse.

As far as I'm aware the same pattern exists in a lot of languages and dialects, and it's possible to define formal AAVE too.

To your comment that was detected as a duplicate:

>I don't think there is anything "formal" about Standard English.

"Standard" English as she is spoke? No. But a3n was specifically talking about the formal version often called "Standard written English". It is very prescriptive and less flexible, which makes it easier to parse.

> Can you give an example of what you're talking about? Which grammatical rules do you have in mind?

Let's start with "every rule involving punctuation".

I'm not trying to be dense, but I still don't see it. Punctuation helps with identifying boundaries of clauses, whether they're dependent or not, whether a sentence is a question. Is this what NLP struggles with? I don't think so. One of the big unsolved problems is understanding the referents of pronouns (and other proforms, like "I will [do so] too"). Style guides and generally prescriptivism does not help that.

Another ambiguity is parse ambiguities like " I saw the girl with the telescope." What does "formal" English say about that?

That's where AI comes in. Unless there's a human at the ready.

> Do you think written Standard English is less ambiguous than any other English, or written language?

Absolutely. It follows a strict set of formal rules and you can readily find sources to learn those rules. Heck, copyeditors regularly debate the specific nuances of how to consistently write Standard English.

Other dialects (and spoken language) are far less formalized. Where is the AP Stylebook or Strunk & White for AAVE?

Can you give an example where Strunk and White or a style guide vhelp to disambiguate some expression? I can't think of any.

The Oxford comma is the most obvious case I can think of.

This is absolutely one of the reasons. See Google's computer vision system classifying black folks as gorillas: http://blogs.wsj.com/digits/2015/07/01/google-mistakenly-tag...

Partly that, but my guess would be that it's because it's largely trained from written English (not the sort you see on Twitter and in SMS), and written English is usually one of the standard varieties, and the standard varieties bear little resemblance to AAVE.

I would guess the same problem occurs with other dialects which are sufficiently different from the standard.

Do you know of any AI's being written by predominately black teams?