Can you elaborate on what kind of training data was used here? I'm curious.
It's like the proverbial witch's pot where they put everything in and out comes the magic.
But it also shows it's not very good on any of those tasks. In any case, machine translation, despite its great popularity as a natural language processing task, is another AI task for which we do not have good metrics.
>> Just by a simple example of translation, it understands the task and does translation. By another example, it can do math, and by another one it can do reasoning, or write react apps.
I think in general, there's a tendency to overestimate the capabilities of GPT-3 for various reasons. Speaking of "understanding" and "reasoning" is really not justified.
On the one hand, it's a language model and most of the tasks it's applied to are tasks for which we don't have very good metrics or benchmarks. Like I say in my earliest comment above, natural language understanding metrics are very bad at measuring "understanding" and we don't even have a commonly agreed definition of what that means. Basically, many benchmarks are defined as classification tasks, e.g. with multiple choice questions supposedly testing a model's understanding, but without any way to ensure that a system is not overfitting to statistical regularities in the dataset - and, indeed, language models have often been shown to do exactly that (e.g. see [1]).
On the other hand, OpenAI very aggressively promotes its systems (not just GPT-3) to users outside of AI research and those users have no way to perform a systematic evaluation of such claims, so they are left with good old eyballing [2] of stuff like language generation or translation, etc. It's all too easy for such users to be impressed by a few hand-picked examples provided by OpenAI itself, or by other users who also don't have the capacity for systematic evaluation (and who hand-pick their results out of undue excitement, rather than for any other reason).
The result is that there is a public perception that OpenAI's language models are much better than they really are. If memory serves, OpenAI made a very big todo about how GPT-2 was so good it was dangerous, etc. Well, now we have GPT-3 which is reportedly even more better- but it's served as an API. Doesn't sound that dangerous and it all sounds a lot more as hype than actual progress.
____________
[1] Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
https://arxiv.org/abs/1902.01007
[2] I keep saying that word, but it's actually semi-formal terminology. There was a paper about evaluating the results of grammar induction algorithms that used it. I'll see if I can find it.
I believe such a situation would generate much less debate in software engineering: "my program passes all my unit tests, but it still crashes". Well, yes. Your program passes all your unit tests, because your unit tests are missing the point, not because your code works.
This means we could potentially dig out thousands of uses out of it by carefully crafting triggers. It's a more general kind of tool than what we're accustomed to work with. It opens a new direction that might become a large field in five years, if they manage to make it run on a regular computer.
<rant>I imagine a GPT-3 like model coupled with search (having Google in an internal loop), trained with multimedia - images, videos, audio, papers, code so it has grounded concepts, and being able to generate text, images, video and code as output. Then I imagine having curated thousands of tasks from the community and added in the training set so it becomes much more efficient, and having all these capabilities exposed as a general AI library. It will be able to work with any modality and you will be able to describe your task in natural language. All of these are possible today. GPT-3 has shown the power of learning to predict on 500B tokens.</>