Openai states themselves that they trained on "40GB of internet text"

https://openai.com/blog/better-language-models/

Your link is for GPT-2, GPT-3 used much, much more data.

GPT-3 was trained in part on "books2" corpus which is not public but seems to basically be the same thing as this: 200k books * 100k words per book on average * ~3 token per words = 60B tokens, books2 is 55B tokens so it checks out.

The total amount of tokens that GPT-3 was trained on from all sources is a combined 500B tokens, this is merely 10% of what they have.

https://arxiv.org/pdf/2005.14165.pdf

Obviously our wetware is optimized for it and we aren't true blank slates, but the enormous magnitude of the discrepancy makes me think we're on the wrong path.

The ironic thing is that ML became popular because we didn't have the technology back then to properly model actual neuronal activity. Now we do, but ML is so established that modeling "true" intelligence is no longer de jure.

So, GPT? I think that's probably a form of intelligence too, but not one that's particularly similar to our own. I also doubt that we'll have much luck getting it to ever perform at near-human levels, when you consider the data and power requirements. I think it's not simply a matter of silicon vs wetware; I think the GPT approach is substantially more different from our own than the hardware it runs on. It's a form of intelligence but that doesn't mean it is like our own.

https://www.technologyreview.com/2020/08/22/1007539/gpt3-ope...

That's not a fair comparison. The human neural network is not trained _from scratch_. There is a very good fundamental structure we inherited through millions of years of evolution.

It is worth noting that the training data used for training humans has been undergoing it's own optimization process for a while too (arguably for about 40ky, since the Upper Paleolithic Revolution, which is roughly when cultural evolution started taking over). In AI adjusting the training data is called Curriculum Learning. Right now most curriculum learning is done just by adjusting the order of training samples, rather than creating samples specifically optimized to facilitate learning (although GANs might be considered a Socratic approach to the latter, if you squint at it).

Instead, as you pointed out, it's just proof of how efficient our wetware is.

books3 may be 10%, but The Pile is building the rest:

https://twitter.com/arankomatsuzaki/status/13204141418954874...

https://github.com/EleutherAI/The-Pile

https://www.eleuther.ai/get-involved

https://media.discordapp.net/attachments/735217892517216366/...

https://media.discordapp.net/attachments/735217892517216366/...

I just have a minor gripe with saying that "now we can train world class GPT model" thanks to that, as you said it's just one piece, and as a typicial HNist I had to point it out :).

But after spending roughly one year acquiring knowledge related to this work, I feel I can say with a fairly high degree of certainty that this dataset alone is enough to train a model that will achieve "world class" status in some area. Writing books, perhaps.

Which part of my logic do you feel is mistaken, and why? I am actually quite interested to hear thoughts from someone who is very pedantic about such things.