Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by deepsquirrelnet on Wednesday November 01, 2023 @06:08PM
(#38106008)
I’ve been impressed with “fuzzy” deduplication at this data scale. I’ve used minhash and networkx for small amounts of data, but I really appreciated the write up on your GitHub about how you implemented it for this dataset.
[ Parent
| on HN ]
There are no comments on this comment yet.