News for hackers. Stuff that matters.
Re:PaLM 2 Technical Report [pdf] by FanaHOVA on Thursday May 11, 2023 @12:25AM (#35897467)
They don't specify, but if you're generally curious you should look into mC4, RedPajama, The Stack, etc as they are the foundation of most training sets.
Re:PaLM 2 Technical Report [pdf] by simonw on Thursday May 11, 2023 @12:46AM (#35897647)
I've spent quite a bit of time exploring RedPajama! https://simonwillison.net/2023/Apr/17/redpajama-data/