Agreed! But, it's important not to confuse what's possible via 'human-in-the-loop' 'active-learning' with what's possible from an algorithm on a fixed training corpus.

Sometimes GPT-like models are portrayed as the latter – a highly automated, reproducible process – while this article makes a pretty strong case that the responses from OpenAI's public-facing interface get rapidly improved by a staff of dozens of contractors.

It's not surprising that a staff of 40 humans, given a few days time to consider & compose, can prepare human-quality answers to arbitrary questions!

I think, ultimately, what matters is how the labeling and corrective actions falloff over time.

But reasonable people will notice that information is streaming In faster than any single person can code

In this example, it seems they fell off immediately, because the responses were only coded in response to questions asked in the way the author pre-published, and did not appear when the same question was asked in a slightly different way, which the GPT-3 authors did not have the opportunity to prepare for well ahead of time

It is the (apparent?) ability to make inferences that makes GPT-3 look impressive. Take that away, and it looks more like a demonstration of the banality of everyday chatter than a significant development towards AI.

Ah so when OpenAI codes fizzbuzz, it's returning human tweaked fizzbuzz from the lookup table? https://twitter.com/sama/status/1503820489927495682

No, I didn't say anything even a lightyear close to that.