Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by civilitty on Monday October 30, 2023 @10:17PM
(#38079353)
> Seems like we are hitting that number pretty quick?
I don't think we're even close. Libgen's nonfiction archive alone is over 32 terabytes. Total size last year was over 120 terabytes. Between that, SciHub, and the internet, there's probably orders of magnitude more tokens out there.
I don't think we're even close. Libgen's nonfiction archive alone is over 32 terabytes. Total size last year was over 120 terabytes. Between that, SciHub, and the internet, there's probably orders of magnitude more tokens out there.