Re:Tempering Expectations for GPT-3 and OpenAI’s API
by neurologic on Monday July 20, 2020 @12:17AM
(#23894610)
The novelty of GPT3 is its few shot learning capabilities. GPT3 shows a new, previously-unknown, and, most importantly, extremely useful property of very large transformers trained on text -- that they can learn to do new things quickly. There isn't any ML researcher on record who predicted it.
Re:Tempering Expectations for GPT-3 and OpenAI’s API
by MiroF on Monday July 20, 2020 @01:54PM
(#23900233)
> There isn't any ML researcher on record who predicted it.
That's just absurd - this was an obvious end-result for LM. NLP researchers knew that something like this was absolutely possible, my professor predicted it like 3 years ago.
It is nothing more than one big transformer. At a technical level, it does nothing impressive, apart from throw money at a problem.
So in that sense, having already been impressed at Transformers and then ELMO/BERT/GPT-1 (for making massive pretraining popular). There is nothing in GPT3 that is particularly impressive outside of Transformers and massive pre-training, both of which are well known in the community.
So, yeah, I am very impressed by how well transformers scale. But, idk if I'd give OpenAI any credit for that.