Re:OpenAI's GPT-3 may be the biggest thing since Bitcoin
by vladTheInhaler on Monday July 20, 2020 @04:13AM
(#23895762)
My understanding is that GPT-2 was actually trained on a dataset that was designed to avoid those pitfalls. They followed all the links posted to Reddit that had more than a couple karma, under the theory that the content was at least slightly interesting to some actual humans, as opposed to a giant blob of search keywords or what have you.
I would like to see a GPT model where training data is weighted by credibility / authority (e.g. using Pagerank).