And meaningless distinction of the year award goes to..
"It's not real [insert property]" is not an intelligent argument. By all means, divine the way to distinguish results of the two. After all, what kind of important distinction can't be distinguished by results?
>Further it didn’t happen over night but through successive improvements
The only difference between GPT-3 and GPT-2 was scale. They didn't even change the tokenizer until 4. There were no "successful improvements" to smoothen the massive gap in capabilities between the two. So to say that was expected just shows more lack of knowledge here.
"Other than that, how was the play, Mrs. Lincoln?"
GPT-2 is two orders of magnitude smaller. In terms of forebrain neuron count this is the difference between a human and an elephant shrew or budgerigar. It was absolutely expected by reasonable people that 2 OOMs of scaling will provide a qualitative jump.
When GPT-3 was released, it was by far the largest artificial neural network ever trained. And I mean by far. Now there wasn't any big jump in hardware capabilities to spur this sort of gulf. It wasn't a case of "Oh now we can train a very large model"
So Want to know why there was such a gap ? It's because most researchers assumed the models would overfit the data or display diminishing returns long before 175b.
Brain neurons are not comparable to ann parameters.
They are not machines that reason, they are approximators. It’s all just token matching based on data we’ve fed it. Further it didn’t happen over night but through successive improvements and at no point was the next improvement considered some infeasible thing.
Seems to me maybe you’ve bought into the hype here, and are confusing that with reality.