Re:Self-supervised learning: The dark matter of intelligence
by 3abiton on Monday September 18, 2023 @06:01PM
(#37562686)
In light of LLMs it seems relevant.
Re:Self-supervised learning: The dark matter of intelligence
by thelastparadise on Monday September 18, 2023 @06:09PM
(#37562780)
Especially in light of the releases of LLaMA and v2.
Re:Self-supervised learning: The dark matter of intelligence
by p1esk on Monday September 18, 2023 @09:04PM
(#37564455)
I have no idea what you mean. The article is mainly about dealing with uncertainty when trying to predict visual information. LLMs have no such problem.
Re:Self-supervised learning: The dark matter of intelligence
by lukeinator42 on Monday September 18, 2023 @10:00PM
(#37564826)
LLMs are using self-supervised learning and the article talks about uncertainty when trying to predict missing information in the NLP domain and how it compares to the visual domain (and why it works so well for the NLP domain).
Re:Self-supervised learning: The dark matter of intelligence
by isaacfung on Monday September 18, 2023 @11:48PM
(#37565396)
This article was written in 2021 when masked language models have been successfully applied in nlp(Bert, word2vec, glove, etc). However at the time, it was unclear how the same technique could be applied to vision tasks because unlike language which has a limited vocab, you can't explicitly assign a probability to every possible image. Since then researchers have already made significant progress with techniques like contrastive learning(simclr), self distillation (BYOL, DINO), masked image models, etc. A cookbook of self-supervised learning is a good source to learn more about this topic.
https://arxiv.org/abs/2304.12210
Re:Self-supervised learning: The dark matter of intelligence
by imjonse on Tuesday September 19, 2023 @12:42AM
(#37565673)
SimCLR and others are older than 2021, BYOL is even mentioned in the blogpost. But your link indeed points to a more comprehensive overview.
Re:Self-supervised learning: The dark matter of intelligence
by isaacfung on Tuesday September 19, 2023 @01:50AM
(#37566016)
You are correct that SimCLR and BYOL were released one year earlier. Sorry I worded it poorly. By "at the time", I meant the period of time when masked language models just found success in NLP.
Self-supervised learning: The dark matter of intelligence - https://news.ycombinator.com/item?id=26431507 - March 2021 (74 comments)