Re:Visual Introduction to Self Supervised Learning
by alleycat5000 on Friday February 28, 2020 @02:18PM
(#22445466)
Some really interesting work lately on "contrastive" learning, where the accuracy is really getting on par with supervised learning, e.g. https://arxiv.org/abs/2002.05709
Re:Visual Introduction to Self Supervised Learning
by yazr on Friday February 28, 2020 @03:32PM
(#22446063)
A fully illustrated article [1]
And Lilian Weng blog on self-supervision [2]
.. CPC is .. translating a generative modeling problem to a classification problem... uses cross-entropy loss to measure how well the model can classify the “future” representation amongst a set of unrelated “negative” samples...
Re:Visual Introduction to Self Supervised Learning
by Voloskaya on Friday February 28, 2020 @05:41PM
(#22447093)
(one variant of) The task is: Given a crop of an image (or a short audio snippet etc.), can you find the matching crop that also comes from the same image from a set containing a lot of negative samples (crops of others, unrelated images)?
To succeed, the encoder needs to be able to extract the underlying, useful information (called slow features) contained in the patch and discard the noise as this will make the retrieval process much easier.
This yields an encoder that gives pretty good representations of your inputs and you can then finetune some additional layers on top of it for your final task.
Re:Visual Introduction to Self Supervised Learning
by trash3 on Friday February 28, 2020 @05:14PM
(#22446904)
So instead of image annotation, self-supervised learning performs image manipulation to train a model. Then what? Is this network then piped into the original task at hand which would have required human annotations or is it simply for these made up tasks?
Re:Visual Introduction to Self Supervised Learning
by Voloskaya on Friday February 28, 2020 @05:45PM
(#22447121)
You then add a few additional layers on top, and you train those new layers in a classic supervised way.
But because a lot has already been learned you need way less labels.
Re:Visual Introduction to Self Supervised Learning
by ivalm on Saturday February 29, 2020 @02:06AM
(#22449384)
Two main things you can do:
1) Transfer learning -- start with self supervised model and either fine tune parameters or freeze parameters + add another layer to train your task (with way fewer params/necessary labels since you already learned about the input distribution)
2) Nearest neighbor/clustering -- no need to label all classes, simply fetch similar examples (eg find semantically similar sentences in a corpus).
Re:Visual Introduction to Self Supervised Learning
by contravariant on Friday February 28, 2020 @08:03PM
(#22447883)
Optimistically if a self-supervised algorithm is capable of understanding a concept then it shouldn't need all that many examples to make it useful. Ideally you could just show it what cats look like (with just a couple of examples) and ask it to find more of them.
And Lilian Weng blog on self-supervision [2]
.. CPC is .. translating a generative modeling problem to a classification problem... uses cross-entropy loss to measure how well the model can classify the “future” representation amongst a set of unrelated “negative” samples...
[1] https://ankeshanand.com/blog/2020/01/26/contrative-self-supe...
[2] https://lilianweng.github.io/lil-log/2019/11/10/self-supervi...
To succeed, the encoder needs to be able to extract the underlying, useful information (called slow features) contained in the patch and discard the noise as this will make the retrieval process much easier.
This yields an encoder that gives pretty good representations of your inputs and you can then finetune some additional layers on top of it for your final task.
1) Transfer learning -- start with self supervised model and either fine tune parameters or freeze parameters + add another layer to train your task (with way fewer params/necessary labels since you already learned about the input distribution)
2) Nearest neighbor/clustering -- no need to label all classes, simply fetch similar examples (eg find semantically similar sentences in a corpus).