News for hackers. Stuff that matters.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by applgo443 on Tuesday October 31, 2023 @01:15AM (#38080612)
If it's 5 common crawls, isn't data across multiple common crawls mostly similar?
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by zhangce on Tuesday October 31, 2023 @01:38AM (#38080718)
We did an exact dedup across all 84 dumps; there are 100T tokens before this exact dedup, and 30T tokens after. If we do further fuzzy dedup (we have simhash signatures pre-computed for different similarity level), this can potentially be reduced further.

There are quite a lot redundancies across dumps; but also a lot of unique/distinct documents