News for hackers. Stuff that matters.

A comment on Google's Live from Paris Event Private/Deleted Immediately. Parent comment

Re:Google's Live from Paris Event Private/Deleted Immediately by CSMastermind on Wednesday February 08, 2023 @12:15PM (#34711230)
Around of quarter of GPT's training data was Reddit so in some sense it's already a Reddit response generating API.
[ Parent | on HN ]
Re:Google's Live from Paris Event Private/Deleted Immediately by Cipater on Wednesday February 08, 2023 @02:45PM (#34713775)
This can't be right. Any place I can read about this?
[ Parent | on HN ]
Re:Google's Live from Paris Event Private/Deleted Immediately by CSMastermind on Wednesday February 08, 2023 @03:48PM (#34714863)
https://arxiv.org/pdf/2005.14165.pdf

WebText and WebText2 referenced in their papers are corpuses based on Reddit submissions which had a 22% weight in their training model.

https://openwebtext2.readthedocs.io/en/latest/

This is larger than Wikipedia (3% weight) or either of their two book corpuses (8% each).

The only other data included was a filtered set from Common Crawl (weighted 60%).

---

I was imprecise with my language before but hopefully that at least provides some clarity.

[ Parent | on HN ]