In contrast, the weights are the preferred form for modification, even for the company that built it. They only very rarely start a brand new training run from scratch, and when they do so it's not to modify the existing work, it's to make a brand new work that builds on what they learned from the previous model.
If the company makes the form of the work that they themselves use as the primary artifact freely available, I'm not sure why we wouldn't call the work open.
Preferred is obviously not a particularly strong line.
If someone ships object code for a bunch of stable modules, and only provides the source for code that’s expected to be where changed, is that really open?
“Preferred” gets messy quick. Not sure how that can be defined in any consistent way. Models are going to get more & more complex. Training with competitive models, etc.
I think you either have it all, or it isn’t really open. Or only some demarked subset is.
I don't think your argument holds any water.
Now that I've beat my head against this issue for a while, I think it's best summed up as: weights are a binary artifact, not source of any kind.
No one trains an existing model from scratch, even those who have access to all of the data to do so. There's just no compelling reason to retrain a model to make a change when you have the weights already—fine tuning is preferred by everyone.
The only people I've seen who've asserted otherwise are random commenters on the internet who don't really understand the tech.
> ...fine tuning is preferred by everyone
How do you know this? Did you take a survey? When? What if preferences change or there is no consensus?
> The only people I've seen who've asserted otherwise are random commenters on the internet who don't really understand the tech.
There are lots of things that can be done with the training set that don't involve retraining the entire model from scratch. As a random example, I could perform a statistical analysis over a portion of the training set and find a series of vectors in token-space that could be used to steer the model. Something like this can be done without access to the training data, but does it work better? We don't know because it hasn't been tried yet.
But none of that really matters, because what we're discussing is the philosophy of open source. I think it's a really bad take to say that something is open source because it's in a "preferred" format.
Preferred form and under a free license. Llama isn't open source, but that's because the license has restrictions.
As for if it's a bad take that the preferred form matters—take it up with the GPL, I'm just using their definition:
> The “source code” for a work means the preferred form of the work for making modifications to it.
> By the same logic, the comparison between a compiled artifact and weights fails because the weights are not "compiled" in any meaningful sense.
To me the weights map to assembly and the training data+models map to source code+compilers. Sure, you can hand me assembly, and with the assembly I may be able to execute the model/program, but having it does not mean that I can stare at it and learn nor modify it with a reasonable understanding of what's going to change.
I got to add that the situation feels even worse than assembly, because assembly, hand-coded or mutilated by an optimizing compiler still does something very specific and deterministic, but the weights on the model makes things equivalent to programming without booleans, but seemingly random numbers and checking for inequalities to get a binary decision.