News for hackers. Stuff that matters.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by gardnr on Monday October 30, 2023 @11:40PM (#38079993)
Anyone know how large it is?

They state the 1 trillion token dataset is 5TB.

Is it safe to assume this is 5TB * 30 = 150TB?

The code in the HuggingFace repo downloads data from url base: https://data.together.xyz/redpajama-data-v2/v1.0.0

https://huggingface.co/datasets/togethercomputer/RedPajama-D...

Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by zhangce on Tuesday October 31, 2023 @12:45AM (#38080445)
It is around 100TB (84 CommonCrawl dumps, roughly 1TB per dump)
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by mauriceweber on Tuesday October 31, 2023 @03:51PM (#38090495)
yes, small clarification: the 1TB per dump refers to the head+middle partition of the dataset and includes the text documents and the quality signals. There is another ~700GB for the minhash signatures and 1-1.5TB for the documents in the tail split.