Re:Snowflake Arctic Instruct (128x3B MoE), largest open sour...
by furyofantares on Wednesday April 24, 2024 @03:15PM
(#40148345)
There's no perfect analogy. It's far easier to usefully modify the weights of a model without the training data than it is to modify a binary executable without its source code.
I'd rather also have the data for sure! But in terms of what useful things I can do with it, weights are closer to source code than they are to a binary blob.
I get that there may be some training data that is proprietary and cannot be released. But in those scenarios, it would still be good to know what the data is, how it was curated or filtered (this greatly affects LLM performance), how it is weighted relative to other training data, and so forth. But a significant portion of data used to train models is not proprietary and in those cases they can simply link to that data elsewhere or release it themselves, which is what others have done.