If you can't reproduce the model then it's not open-source

Re:If you can't reproduce the model then it's not open-source by cpeterso on Wednesday January 17, 2024 @04:27PM (#39033733)

Are there any true open-source LLM models, where all the training data is publicly-available (with a compatible license) and the training software can reproduce bit-identical models?

Is training nondeterministic? I know LLM outputs are purposely nondeterministic.

[ Parent | on HN ]

Re:If you can't reproduce the model then it's not open-source by Trapais on Thursday January 18, 2024 @10:52AM (#39043247)

>Are there any true open-source LLM models, where all the training data is publicly-available (with a compatible license)

Mamba has a version, trained on publicly available SlimPajama. RedPajama-INCITE was trained on non-slimmed version of the dataset(it's only one dataset).

I'm not sure if training scripts are available.

Pythia definitely has scripts. However it was trained on the pile, so you have to find books3 on your own.

Also I believe LLM360 is an explicit attempt to do it with llama.

>Is training nondeterministic?

Correct. Torch documentation has a section on reproducibility of a training.

[ Parent | on HN ]