News for hackers. Stuff that matters.
Re:OpenAI API by d_burfoot on Thursday June 11, 2020 @09:11PM (#23494964)
In NLP there is a very clear and powerful new paradigm: train a HUGE language model using vast amounts of raw text. Then to solve the problem of interest, either fine-tune the model by training on your specific dataset (usually quite small), or 0/1-shot the learning somehow.

The crucial question is : is this paradigm viable for OTHER types of data?

My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.

I'm especially optimistic that this paradigm will work for image streams from autonomous vehicles. Classic supervised learning has proved to be difficult if not impossible to get to work for AV vision, so the new paradigm could be a game-changer.

Re:OpenAI API by gwern on Thursday June 11, 2020 @09:16PM (#23494983)
> My hypothesis is YES. If you train a HUGE image model using vast quantities of raw images, you will then be able to REUSE that model to work for specific computer vision problems, either by fine-tuning or 0/1-shotting.

This has been demonstrated for many years, it's not news. Many of the SOTAs like BiT require pretraining on JFT-300M, or Instagram, or what have you.

Re:OpenAI API by canjobear on Friday June 12, 2020 @02:15AM (#23496537)
The pretraining approach was used in vision for years before it was successful in NLP.
Re:OpenAI API by julien_c on Friday June 12, 2020 @03:09AM (#23496820)
Not really on unsupervised/self-supervised data though, right?

(nor on the same scale of corpora, as far as I can tell)