News for hackers. Stuff that matters.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by ipsum2 on Monday April 17, 2023 @04:09PM (#35605908)
Huh? There's plenty of open source LLMs. Pythia, GPT-NeoX, GPT-J, GPT-2, BLOOM-176, are ones I can think of off the top of my head. Pythia is the best performing one IIRC.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by bradknowles on Monday April 17, 2023 @08:32PM (#35608517)
Hmm. Would you be able to combine these LLMs? Or are they already supersets of each other?
Re:RedPajama: Reproduction of Llama with Friendly License (S... by MacsHeadroom on Monday April 17, 2023 @09:37PM (#35608933)
Pythia was trained on only 300B tokens and is pretty dumb compared to LLaMA.

Pythia 13B is worse than LLaMA-7B and requires double the resources.

Re:RedPajama: Reproduction of Llama with Friendly License (S... by Tepix on Tuesday April 18, 2023 @11:12AM (#35615059)
Not all use cases need GPT-4 level performance. I'd argue that even LLaMA-7B is quite limited. Also, new and improved models are being released all the time.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by lhl on Tuesday April 18, 2023 @09:08AM (#35613494)
I started keeping a list btw, there are about 20 completely open (Apache 2.0, BSD, MIT, CC-BY) 1B+ parameter foundational LLMs at the moment: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...