On the contrary, there is every reason to assume that GPT-3 cannot significantly improve its results on those tasks with extensive fine tuning. Because, if GPT-3 could significantly improve its performance on those tasks with extensive fine-tuning, the OpenAI paper would be reporting those results (edit: OpenAI sure has the time and money to finetune their model).
As we all know by now, there is strong bias in never reporting negative results in machine learning as in other research fields, so we can be reasonably certain that if there is an obvious experiment to perform and that experiment is missing from a paper, it was attemtped and the results were poor.
> "OpenAI sure has the time and money to finetune their model"
This model is absolutly gigantic. In term of training it's a nightmare. I am pretty sure they don't have the capacity to train 10 of them in parallel. So finetuning on all the downstream task needs to be done basically in serial and takes forever. They might have a lot of money, but time is as important for them than for anyone else, their lab isn't at the edge of black hole.
If what you are saying is true I am pretty sure they would have reported it because it would be an extremely interesting and important result in my opinion. If finetuning gave you no advantage over a few shot settings, it would basically mean that the model already know everything there is to know about the task just from it's pretraining and any additional training is useless as the model is not learning anything.
Finally, given the pre-training curves with various model sizes, we clearly haven't reached saturation there, there is no indication anywhere that we have reached saturation on downstream task.
So, for me the far more likely explanation is that fine-tuning on downstream tasks is indeed very costly (time and/or money) even by their standard and isn't even on topic for this paper.
Outside of academia I see this thing very often. "Oh, I'm sure if it was easy to do that, they'd have done it". No. This is not how a piece of research work is evaluated, not even a piece of work in deep learning, a field that has abandoned all pretensions to science in recent years.
My personal advice (speaking as someone who has been attacked by the hyenas and paid my pound of flesh) is that one should always demand the highest standard of proof for any claim in a research paper. That, if one really wishes to know what's going on. Intellectual curiosity and scientific wonderment should not result in gullibility.
Do you not see the cognitive dissonance? You were precisely claiming that because it should be easy to do they most likely have done it, and if they didn't report results it's because it failed.
You are making this unsupported claim with 0 evidence to back you up.
In any case, it's an obvious thing to do and there's no obvious explanation of why they didn't, given that they could.
It is very good in all those tasks. In the paper all the numbers comparing GPT-3 with other models on those benchmark are comparing GPT-3 in a zero/one/few shot setting (so no gradient step), against the previous state of the art finetuned for hours or days on the specific task with millions of gradients steps. If you had time and money to finetue GPT-3 on the specific task there is every reason to believe the gap would be huge.
This is the big thing about GPT-3, the promise of not needing to finetune anymore. This is huge in term of productivity but also because it allows you to use the model in settings for which there are basically no dataset available to finetune on.