The whole schtick of GPT-3 is the insight that we do not need to come up with a better algorithm than GPT-2. If we dramatically increase the number of parameters without changing the architecture/algorithm its capabilities will actually dramatically increase instead of reaching a plateau like it was expected by some.
Edit: Source https://www.gwern.net/newsletter/2020/05#gpt-3
"To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI."
Any time you see an AI promo with really big numbers claiming to be close to replicating human cognitive functions, just remember that the human brain has 100 trillion synapses. So much of our discourse around AI right now is nothing more than chimpanzees scribbling on a wall with a crayon and calling it a "self portrait".
Googling around, it looks like most neural networks have somewhere in the neighborhood of tens of thousands of parameters. If nothing else GPT-3 is much, much bigger than most of its peers.
We can see a very strong correlation between brain size, and overall intelligence within the animal kingdom. The larger the brain, the smarter the animal.
Effectively, GPT-3 has a bigger brain.
This without including animals such as parrots and octopuses.
Pretty sure it holds for hominids, though.
In general size of brain is proportional to body mass, more sensors bigger brain, has nothing to do with intelligent effective behavior, to large extent, arguably.
To best of my knowledge there is no known method that would estimate minimum amount of neurons even for simple problems, let alone complex one, however there is convergence of some form cerebral cortex between species, but octopi break this model to large extent (might we wrong here). Things get even more complicated when you account for space between individual neurons.
I don't know much about AI, though I do know about programming, and to me this vaguely smells like "our program is so great because it has 1 million lines of code!"
Does the number of parameters, dimensions, etc, really have anything to do with how breathtaking and marvelous something like this is?