News for hackers. Stuff that matters.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by omneity on Monday April 17, 2023 @01:41PM (#35603988)
An actually open source LLM would be a game changer. We might need a new license that englobes model usage and training, something GPL-like whereby distributing a retrained model requires contributing data back or making it public, but not if you use it privately.

This will definitely accelerate progress in LLM research, productization and safety. Alpaca, vicuna, gpt4all and others are sporadic repesentations of this that could become a continuous improvement process were the LLM and its license truely open source.

An interesting possible side effect of a GPL-like license is that AIs are unlikely to be trained on private data, the usual moat that big tech wouldn't want/just can't make public if it were to use those GPL-like licensed models.

Re:RedPajama: Reproduction of Llama with Friendly License (S... by jupp0r on Monday April 17, 2023 @02:28PM (#35604570)
As with original GPL, this would be almost useless in a commercial context.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by e12e on Monday April 17, 2023 @02:33PM (#35604631)
There are commercial devices that ship with a Linux kernel?
Re:RedPajama: Reproduction of Llama with Friendly License (S... by ijustlovemath on Monday April 17, 2023 @03:13PM (#35605149)
I think they mean in terms of enforcement when there's a violation
Re:RedPajama: Reproduction of Llama with Friendly License (S... by Bjartr on Monday April 17, 2023 @04:07PM (#35605877)
Basically every Android device for starters.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by sp332 on Monday April 17, 2023 @04:57PM (#35606517)
Using a Linux kernel doesn't mean you have to make your whole project GPL, unless your project is specifically kernel code.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by wongarsu on Monday April 17, 2023 @06:04PM (#35607237)
Neither would the proposed model license. Just like the kernel's GPL stops at the userspace boundary, the proposed license would only cover the model definition and weights.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by jupp0r on Monday April 17, 2023 @06:25PM (#35607472)
But do they train the Linux kernel with their customers data?
Re:RedPajama: Reproduction of Llama with Friendly License (S... by ipsum2 on Monday April 17, 2023 @04:09PM (#35605908)
Huh? There's plenty of open source LLMs. Pythia, GPT-NeoX, GPT-J, GPT-2, BLOOM-176, are ones I can think of off the top of my head. Pythia is the best performing one IIRC.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by bradknowles on Monday April 17, 2023 @08:32PM (#35608517)
Hmm. Would you be able to combine these LLMs? Or are they already supersets of each other?
Re:RedPajama: Reproduction of Llama with Friendly License (S... by MacsHeadroom on Monday April 17, 2023 @09:37PM (#35608933)
Pythia was trained on only 300B tokens and is pretty dumb compared to LLaMA.

Pythia 13B is worse than LLaMA-7B and requires double the resources.

Re:RedPajama: Reproduction of Llama with Friendly License (S... by Tepix on Tuesday April 18, 2023 @11:12AM (#35615059)
Not all use cases need GPT-4 level performance. I'd argue that even LLaMA-7B is quite limited. Also, new and improved models are being released all the time.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by lhl on Tuesday April 18, 2023 @09:08AM (#35613494)
I started keeping a list btw, there are about 20 completely open (Apache 2.0, BSD, MIT, CC-BY) 1B+ parameter foundational LLMs at the moment: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...
Re:RedPajama: Reproduction of Llama with Friendly License (S... by buzzscale on Monday April 17, 2023 @06:31PM (#35607521)
Dolly 2.0 is fully open, Apache License and the tuning dataset is employee generated:

https://www.databricks.com/blog/2023/04/12/dolly-first-open-...

Re:RedPajama: Reproduction of Llama with Friendly License (S... by gumballindie on Monday April 17, 2023 @07:23PM (#35607960)
The solution is simple. We need an updated GPL license that states that the code cant be used in training ais unless the data model is also open source. A coordinated update of all major open source projects and the issue is sorted as it will force the ai folks to open source their models. Or else they’ll have to stick with generating funny cat pictures.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by jacooper on Monday April 17, 2023 @08:26PM (#35608478)
The problem isnt the license, the problem is it being fair use or not. If its fair use the license is irrelevant.
Re:RedPajama: Reproduction of Llama with Friendly License (S... by gumballindie on Tuesday April 18, 2023 @01:32AM (#35610311)
That has an even easier fix: no content no problem.