Re:Google's Live from Paris Event Private/Deleted Immediately
by Cipater on Wednesday February 08, 2023 @02:45PM
(#34713775)
This can't be right. Any place I can read about this?
[ Parent
| on HN ]
Re:Google's Live from Paris Event Private/Deleted Immediately
by CSMastermind on Wednesday February 08, 2023 @03:48PM
(#34714863)
https://arxiv.org/pdf/2005.14165.pdf
WebText and WebText2 referenced in their papers are corpuses based on Reddit submissions which had a 22% weight in their training model.
https://openwebtext2.readthedocs.io/en/latest/
This is larger than Wikipedia (3% weight) or either of their two book corpuses (8% each).
The only other data included was a filtered set from Common Crawl (weighted 60%).
---
I was imprecise with my language before but hopefully that at least provides some clarity.