News for hackers. Stuff that matters.
Re:OpenAI's GPT-3 may be the biggest thing since Bitcoin by dannyw on Sunday July 19, 2020 @06:42AM (#23888437)
It could also be a result of training data. If every page is weighted equally, you'd expect SEO spam and even autogenerated content to far surpass high quality content in volume.

I would like to see a GPT model where training data is weighted by credibility / authority (e.g. using Pagerank).

Re:OpenAI's GPT-3 may be the biggest thing since Bitcoin by vladTheInhaler on Monday July 20, 2020 @04:13AM (#23895762)
My understanding is that GPT-2 was actually trained on a dataset that was designed to avoid those pitfalls. They followed all the links posted to Reddit that had more than a couple karma, under the theory that the content was at least slightly interesting to some actual humans, as opposed to a giant blob of search keywords or what have you.