News for hackers. Stuff that matters.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by e12e on Tuesday October 31, 2023 @07:58AM (#38083415)
Nice. I admit I find the language selection a bit uninspired and odd:

> Five languages: English, French, Spanish, German, and Italian

Otoh I'm surprised that when counting first and second language proficiency, German is actually ahead of Japanese....

https://en.m.wikipedia.org/wiki/List_of_languages_by_total_n...

Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by kiney on Tuesday October 31, 2023 @10:09AM (#38085075)
I'm surprise for the low number native german speakers in that list. Thats less than the population of germany alone. And then theres austria, parts of switzerlan, northern italy, western belgium....