500 billion word pieces (it splits words into few-character long pieces), which comes down to 100-200B words, scraped off the internet. It used mostly CommonCrawl as training data.
It's like the proverbial witch's pot where they put everything in and out comes the magic.
Re:Why GPT-3 Matters
by YeGoblynQueenne on Tuesday July 21, 2020 @06:25AM
(#23905744)
Yes, that's an apt metaphor. Magic, indeed.
Can you elaborate on what kind of training data was used here? I'm curious.