Re:PaLM 2 Technical Report [pdf]
by FanaHOVA on Thursday May 11, 2023 @12:25AM
(#35897467)
They don't specify, but if you're generally curious you should look into mC4, RedPajama, The Stack, etc as they are the foundation of most training sets.
[ Parent
| on HN ]
I've spent quite a bit of time exploring RedPajama! https://simonwillison.net/2023/Apr/17/redpajama-data/