News for hackers. Stuff that matters.
Re:Meta Llama 3 by blackeyeblitzar on Thursday April 18, 2024 @01:00PM (#40078327)
Llama is open weight, not open source. They don’t release all the things you need to reproduce their weights.
Re:Meta Llama 3 by mananaysiempre on Thursday April 18, 2024 @01:10PM (#40078440)
Not really that either, if we assume that “open weight” means something similar to the standard meaning of “open source”—section 2 of the license discriminates against some users, and the entirety of the AUP against some uses, in contravention of FSD #0 (“The freedom to run the program as you wish, for any purpose”) as well as DFSG #5&6 = OSD #5&6 (“No Discrimination Against Persons or Groups” and “... Fields of Endeavor”, the text under those titles is identical in both cases). Section 7 of the license is a choice of jurisdiction, which (in addition to being void in many places) I believe was considered to be against or at least skirting the DFSG in other licenses. At best it’s weight-available and redistributable.
Re:Meta Llama 3 by blackeyeblitzar on Thursday April 18, 2024 @04:00PM (#40080109)
Those are all great points and these companies need to really be called out for open washing
Re:Meta Llama 3 by amitport on Friday April 19, 2024 @01:43AM (#40083764)
It's a good balance IMHO. I appreciate what they have released.
Re:Meta Llama 3 by ikurei on Friday April 19, 2024 @06:29AM (#40085074)
I appreciate it too, and they're of course going to call it "open weights", but I reckon we (the technically informed public) should call it "weights-available".
Re:Meta Llama 3 by lumost on Friday April 19, 2024 @01:14AM (#40083626)
Has anyone tested how close you need to be to the weights for copyright purposes?
Re:Meta Llama 3 by tdullien on Friday April 19, 2024 @01:53PM (#40089932)
It's not even clear if weights are copyrightable in the first place, so no.
Re:Meta Llama 3 by ktzar on Friday April 19, 2024 @05:21AM (#40084741)
even if they released them, wouldn't it be prohibitively expensive to reproduce the weights?
Re:Meta Llama 3 by lawlessone on Friday April 19, 2024 @10:38AM (#40087390)
1.3 million GPU hrs for the 8b model. Take you around 130 years to train on a desktop lol.
Re:Meta Llama 3 by iamlearningai on Tuesday April 23, 2024 @08:18AM (#40131064)
Interesting. LLAMA is trained using 16K GPUs so it would have taken around a quarter for them. An hour of GPU use costs $2-$3 so training a custom solution using LLAMA should be atleast $15K to $1M. I am trying to get started with this thing. A few guys suggested 2 GPUs were a good start but I think that would only be good for 10K training samples.
Re:Meta Llama 3 by zingelshuher on Saturday April 20, 2024 @02:05PM (#40099371)
It's impossible. Meta itself cannot reproduce the model. Because training is randomized and that info is lost. First samples a coming at random. Second there are often drop-out layers, they generate random pattern which exists only on GPU during training for the duration of a single sample. Nobody saves them, it would take much more than training data. If someone tries to re-train the patterns will be different, which results in different weight and divergence from the beginning. Model will converge to something completely different. With close behavior if training was stable. LLMs are stable.

So, no way to reproduce the model. This requirement for 'open source' is absurd. It cannot be reliably done even for small models due to GPU internal randomness. Only the smallest trained on CPU in single thread. Only academia will be interested.

Re:Meta Llama 3 by throwaway4good on Friday April 19, 2024 @11:15AM (#40087870)
Which large model projects are open source in that sense? That its full source code including training material is published.
Re:Meta Llama 3 by soccernee on Friday April 19, 2024 @11:55AM (#40088389)
Olmo from AI2. They released the model weights plus training data and training code.

link: https://allenai.org/olmo

Re:Meta Llama 3 by whiplash451 on Sunday April 21, 2024 @03:11PM (#40108315)
Is it really useful to make an LLM open source when it takes millions of $ to train it?

At that scale, open weights with permissive license is much more useful than open source.