News for hackers. Stuff that matters.
Re:Recent Advances in Natural Language Processing by MiroF on Sunday August 16, 2020 @03:10PM (#24180331)
But there's no reason the models are english specific...
Re:Recent Advances in Natural Language Processing by YeGoblynQueenne on Sunday August 16, 2020 @04:21PM (#24180931)
Yes, there is. A "model" is a set of parameters optimisted by some algorithm or system trained on the data in a specific dataset. Thus, a language model trained on a dataset of English language examples is only capable of representing English language utterances, not, e.g. French, or Greek, or Gujarati utterances. Diagrammatically:

  data --> system --> model
What is not necessarily English-specific are the systems used to train different language models, at least in theory. In practice, systems are typically hand-crafted and fine-tuned to specific datasets, to such a degree that most of the work has to be done anew to train on a different dataset.
Re:Recent Advances in Natural Language Processing by mannykannot on Sunday August 16, 2020 @09:16PM (#24182862)
This is very interesting, and I would be interested in learning more about the language-specific accommodations that are made. My minimal understanding of the GPT systems is that they are initially trained on a large corpus of English-language text, and then often, though not necessarily, given few-shot training on example tasks before being tested with questions that are scored.

It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that. One guess is that the training data is annotated with grammatical information, in which case I would wonder if this is just a shortcut, or whether it solves a fundamental problem for such systems. Another guess is that the training set includes disambiguation, but that would seem to render meaningless the results on Winograd schema. (Update: I withdraw this last point, as presumably the Winograd-schema questions themselves are not disambiguated. Disambiguation of the training corpus would be a quite significant language-specific accommodation, if that is what is happening.)

Re:Recent Advances in Natural Language Processing by MiroF on Sunday August 16, 2020 @09:32PM (#24182968)
> It would be unremarkable if the same system would have to be retrained from scratch to work with a different language, but I take it that you mean more than that.

It really is that unremarkable, despite the GPs insinuations to the contrary.

Re:Recent Advances in Natural Language Processing by YeGoblynQueenne on Monday August 17, 2020 @05:20AM (#24185475)
If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc.

In any case, I don't remember anyone ever naming their n-gram models or HMMs etc. The reason for this of course is that training a language model with a deep neural network architecture is anything but unremarkable, not least because the costs in data, human-hours and compute are beyond the reach of most entities other than large corporate teams. If training a new model from scratch were truly "unremarkable" we'd all have our own.

Re:Recent Advances in Natural Language Processing by MiroF on Monday August 17, 2020 @12:46PM (#24188967)
> If training a language model was "unremarkable" we wouldn't have a hypefest everytime anyone releases a new model, with a catchy name no less. BERT, ROBERTA, GPT-2/3/... etc. are remarkable enough for their creators to write papers exclusively for the purpose of announcing the model and describing the architecture used to train it etc.

Actual NLP practitioners are not having a hype-fest every time someone spends a shitton to train a transformer LM on a new dataset. It's definitely cool, but I wouldn't cal it remarkable. What was cool about GPT-3 was the demonstration of the effectiveness of the "few-shot"/"conditioning" approach. I didn't find it surprising, but it was cool to push it to the next level.

> In any case, I don't remember anyone ever naming their n-gram models or HMMs etc

That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point.

As a side remark, you are vastly overestimating the amount of dataset specific tuning that you need - you can often just use the exact same hyperparams and run it on a new dataset - there's been some recent literature showing that a lot of these architectures are actually quite resilient to different datasets and hyperparameter choices and arrive at similar local minima. Sure, you might be off by 1-2% from SOTA accuracy (let's say on a classification task) but in production systems you're generally not shooting for the SOTA that was released in the last 2 months but rather something close-ish to those results.

Honestly, quite confused this is the hill you want to die on.

Re:Recent Advances in Natural Language Processing by YeGoblynQueenne on Monday August 17, 2020 @02:49PM (#24190453)
I'm unsure what you mean by "actual NLP practitioners". In any case does it really make a difference? We've been having GPT-3 related articles on HN and in the techie press for weeks after GPT-3 was published.

>> That's because they are literally different model architectures. The 3-gram model is the same everywhere, it doesn't depend on dataset. A 3-gram "trained" on Russian would still be a 3-gram. Similarly, BERT trained on a German corpus would still be BERT, it wouldn't need a "new catchy name." That's my point.

N-gram models are not "the same everywhere". An n-gram model is a set of probabilities assigned to, well, n-grams. Different sets of n-grams, or probabilities trained from different datasets would be- different. The n-grams would be different and the probabilities would be different and the structure of the model would be different. A Russian n-gram model would represent the probabilities of Russian n-grams, an English one would reprsent the probabilities of English n-grams and so on. The type of model would be the same, an n-gram model; but the model itself would be different. Otherwise, why bother training more models?

Same goes for a model trained with a transformer architecture. BERT is an English language model trained with a transformer architecture and representing the probabilities of tokens in an English corpus. A model trained with a transformer architecture and representing the probabilities of tokens in a German corpus would be, again, a different model. You could also call it "BERT" but that would eventually become very confusing.

>> Honestly, quite confused this is the hill you want to die on.

I'm not dying on any hill. Could you please not do that?

Re:Recent Advances in Natural Language Processing by MiroF on Monday August 17, 2020 @03:23PM (#24190807)
I disagree with how you define "model" to mean the parameters, rather than the form of the relationship between input and output. I don't think every time we make a new estimate of the electron mass, we are creating a "new" Standard Model.

I don't really think it's super worth quibbling over, since I think we're generally on the same page. It's worth noting that BERT trained on different corpora is generally still called BERT (you can look at the huggingface/transformers library, for instance).

I'm sorry that my message came across as more adversarial than intended.

Re:Recent Advances in Natural Language Processing by YeGoblynQueenne on Monday August 17, 2020 @07:08PM (#24193363)
>> I'm sorry that my message came across as more adversarial than intended.

Thanks. I enjoyed our conversation.

Re:Recent Advances in Natural Language Processing by mannykannot on Monday August 17, 2020 @07:41AM (#24186253)
You have hung a lot on the single word “unremarkable”, so perhaps I did not make it completely clear what I thought was unremarkable. The only thing that I meant was that if a given architecture for NLP would require retraining from scratch in order to work with a new language, I would not think that fact would be surprising, or be indicative of some fundamental flaw or limitation in the approach taken.

If, on the other hand, a given architecture could not be retrained to perform well on a different language, or only on a group of related languages, that might be indicative of a fundamental limitation in how far it could be developed, depending on what sort of accommodations have to be made for new languages.

Re:Recent Advances in Natural Language Processing by YeGoblynQueenne on Monday August 17, 2020 @08:48AM (#24186705)
Thanks for clarifying. Part of the reason why training a new model from scratch is so difficult is the amount of dataset-specific tweaking and fine-tuning that goes on and that is rarely discussed in publications. For example, papers willt describe the architecture of a system but will rarely go into much detail about how a particular component was selected over another (why this cost function over the other etc). This fine-tuning is essential to achieving state-of-the-art results and without it you can expect to produce a model that is simply not the same as the original. Certainly in terms of performance metrics.

Basically retraining the same system on a different dataset is a bit like entering the same river in two points in time. Yes, in theory it's "the same" system and only the dataset chnges. In practice, it takes so much architectural tweaking and fine-tuning that it's a different system and you've done almost all the work from scratch. This also is part and parcel of why so much fanfare surrounds the publication of a new model. Because it's really, really hard to train a new model on a new dataset (assuming one wants to get state of the art results).

Re:Recent Advances in Natural Language Processing by MiroF on Sunday August 16, 2020 @09:00PM (#24182774)
I was using model to mean the architecture, ie. the way you're modeling the data flows - not the specific params, which are ofc language specific.
Re:Recent Advances in Natural Language Processing by pc86 on Monday August 17, 2020 @01:54PM (#24189813)
Is it as "simple" as retraining a model on an identical (but translated) dataset, if such a dataset were to exist?
Re:Recent Advances in Natural Language Processing by woodson on Sunday August 16, 2020 @03:29PM (#24180481)
In a certain way, they could be, simply because their structure may work better on the type of language that English is. It may not work as well for languages that exhibit other grammatical patterns (e.g. morphologically rich languages, or those that exhibit superficially more flexible word order which nonetheless conveys information about topic/focus).
Re:Recent Advances in Natural Language Processing by dvduval on Sunday August 16, 2020 @03:36PM (#24180549)
Some languages like Chinese have a large corpus of information that can be studied, and there has been a continuous effort to standardize the language. it wouldn't surprise me if it turns out to be easier than English. I would expect it would be more difficult when you're talking about languages that have regional differences in usage of words, perhaps northern Vietnamese vs. Southern Vietnamese, or the Spanish of Spain versus the Spanish of Mexico.
Re:Recent Advances in Natural Language Processing by east2west on Sunday August 16, 2020 @04:04PM (#24180754)
Chinese has large regional differences, too. Cantonese has different grammar, not just word usage. Modern Chinese grammar is somewhat similar to English while Cantonese follows ancient Chinese, which places verb to the end of sentence.
Re:Recent Advances in Natural Language Processing by DonaldFisk on Sunday August 16, 2020 @06:42PM (#24182001)
Cantonese is, like English and Mandarin, SVO.

E.g. I bought this book: ngóh(S) maáihjó(V) nìbún syù(O).

Contrast with Japanese: watashi wa(S) kono hon o(O) kaimashita(V).

Differences from Mandarin Grammar are listed here: https://en.wikipedia.org/wiki/Cantonese_grammar#Differences_...

Classical Chinese is also SVO: https://en.wikipedia.org/wiki/Classical_Chinese_grammar

Re:Recent Advances in Natural Language Processing by dvduval on Sunday August 16, 2020 @05:53PM (#24181689)
True, and Mandarin would be the better word to use here. Mandarin and Cantonese are for all intents and purposes separate languages. They are more different than English and Spanish.
Re:Recent Advances in Natural Language Processing by DonaldFisk on Sunday August 16, 2020 @07:18PM (#24182229)
Chinese is the written language, based on Mandarin. Mandarin and Cantonese are separate, though closely related, spoken languages. They're a lot closer to each other than English and Spanish (more like French and Spanish), which is why the same written language works for both sets of speakers: most of the vocabulary is shared.
Re:Recent Advances in Natural Language Processing by spacehunt on Sunday August 16, 2020 @11:46PM (#24183713)
This is only because almost all native Cantonese speakers are trained to read and write in Standard Chinese from the very beginning.

Standard Chinese can be spoken in Cantonese, but while understandable, it is never done, in contrast to Mandarin.

In recent years written online conversations (like blogs, Facebook posts, instant messages) by Cantonese speakers are increasingly written in Cantonese instead, and this is slowly spreading to more "formal" media such as books.

Source: I'm a native Cantonese speaker.

Re:Recent Advances in Natural Language Processing by DonaldFisk on Monday August 17, 2020 @12:04PM (#24188499)
Yes, it seems I underestimated the difference: "In spoken Cantonese the frequency of use of Cantonese dialect words is close to 50 percent (Li, et. al., 1995, p236)." (from https://www.aclweb.org/anthology/P98-2238.pdf).

They're still a lot closer than English and Spanish or even French and Spanish (https://www.ezglot.com/most-similar-languages.php).

Re:Recent Advances in Natural Language Processing by yorwba on Sunday August 16, 2020 @06:31PM (#24181932)
If we're going down that path, then it should more specifically be Standard Mandarin, since Standard Mandarin and e.g. Southwestern Mandarin are only partially mutually intelligible like French and Spanish are partially mutually intelligible.
Re:Recent Advances in Natural Language Processing by kingkawn on Sunday August 16, 2020 @05:58PM (#24181722)
All language has significant regional difference if it covers enough regions
Re:Recent Advances in Natural Language Processing by m0zg on Sunday August 16, 2020 @04:16PM (#24180873)
For some of them, there is. Russian (my mother tongue), for example, has the kind of morphology that makes linguists' hair stand on end. Also, verbs and nouns are gendered (and therefore they must agree, and you must know what everything refers to at all times, often pretty far away from where the noun was first mentioned), and there are declensions on various parts of speech, and they must agree as well. And words can be freely formed out of several roots, prefixes, and suffixes. For any of this to be understood or generated, your model has to be able to model all of that, which is much harder than modeling English.
Re:Recent Advances in Natural Language Processing by MiroF on Sunday August 16, 2020 @09:02PM (#24182785)
I think that modern BPE and techniques are more than sophisticated enough to handle things like this.
Re:Recent Advances in Natural Language Processing by m0zg on Sunday August 16, 2020 @11:37PM (#24183654)
Could be, but I've yet to see an example of any NLP task that performs anywhere near as well on Russian as it does on English. Trouble with NLP in general, is that humans, armed with their imperceptible error correction and general intelligence, expect unreasonable things from systems which fundamentally have neither. Error rate in single digit percentages is commonly perceived as terrible, and because of the cardinality of the input space "plausible" errors don't really happen and failure modes are mostly catastrophic.

This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).

Re:Recent Advances in Natural Language Processing by MiroF on Monday August 17, 2020 @01:08AM (#24184184)
> any NLP task that performs anywhere near as well on Russian as it does on English

This has a lot more to do with smaller Russian corpora than anything to do with peculiarities of the Russian language.

> This applies to all languages BTW. There are only very, very few NLP systems that actually work, and they are all in a setting which tolerates a fairly high error rate (e.g. ranking).

I've trained NLP models used on over 15 different languages used in production

Re:Recent Advances in Natural Language Processing by akerro on Sunday August 16, 2020 @03:37PM (#24180556)
I'm 100% sure NLP will do the same mistake with non-English language as face recognition did with faces of black people.

I'm taking bets.

Re:Recent Advances in Natural Language Processing by MiroF on Sunday August 16, 2020 @09:00PM (#24182780)
ie. a dataset/corpus problem?