Most of the times a "fantastic GPT-3 result" is shown, you have to dig a bit and then you'll find out how it was primed[0] and how many different texts they had it generate. Then the one(s) carrying out the experiment go on and pick the most shocking writings. If you read all of the outputs (there are a few articles around that show you 5 or 6 different outputs) you'll see the variations that it took duing those. I understand that 5 or 6 is actually small, to get shocking results they usually go into de dozens of tries.
[0] usually the priming phrases are given, but depending on how much of a snake-oil-salesman the person writing/giving a talk is, they may even hide this part
I think we can get models about 1/100th the size for general use. That's also the main reason Google is developing TPUs.
I don't buy this. OpenAI literally released pricing for GPT-3, so either they grossly miscalculated their cost base (unlikely) or there's some scaling/instability/resourcing issue preventing them from doing so (much more likely).
I think it's telling that they spent the last 6 months on yet another flashy demo (DALL-E) rather than actually productionizing GPT-3. It just feels like constant smoke and mirrors.
The fact that it’s not released (and also the fact that GPT-3 etc are still not publicly available) makes me suspect that these models are far too unstable for actual production use. It’s also why I’m getting a bit tired of these overhyped cherry-picked samples with seemingly nothing solid to ever back it up.