Re:Self Supervised Learning in NLP
by why-el on Sunday May 24, 2020 @04:32PM
(#23294459)
The deepemoji one is fooled by "was my flight delayed? no.". I feel for the computer when it meets that one "do I speak in questions?" person. chuckles.
On a more serious note, Hinton and other alluded to the need to restructure NLP studies as to focus more on the nature of recursion within language, which is basically what Chomsky has been saying for decades. It's interesting to see whether they converge.
Re:Self Supervised Learning in NLP
by labelbias on Monday May 25, 2020 @04:08AM
(#23299080)
I'm wondering if these tasks have a form of bias that decreases the performance. If the model sees only positive examples and no negatives then it is biased on the positive paths of decisions. The moment where one changes the path to be incorrect, the model can't recover from the mistake because there weren't any negative examples during pretraining. There's many words that never follow some words but the model never sees that.
On a more serious note, Hinton and other alluded to the need to restructure NLP studies as to focus more on the nature of recursion within language, which is basically what Chomsky has been saying for decades. It's interesting to see whether they converge.
https://www.aclweb.org/anthology/P19-1455
Also, I always wondered, do those methods work universally on all languages? For example Chinese, Korean and Japanese, with different alphabets.
Some of these methods do. For example, the tasks that power word vectors apply for many languages. https://fasttext.cc/docs/en/crawl-vectors.html
Masked Language Modeling has been applied to learn cross-lingual language models. Look into: https://arxiv.org/abs/1901.07291