Re:Open Source AI Is the Path Forward
by wrs on Wednesday July 24, 2024 @07:07PM
(#41063155)
You can change the tokenizer and build another model, if you can come up with your own version of the rest of the source (e.g., the training set, RLHF, etc.). You can’t change the tokenizer for this model, because you don’t have all of its source.
[ Parent
| on HN ]
There is nothing that requires you to train with the same training set, or to re-do RLHF.
You can train on fineweb, and llama 3.1 will learn to use your new tokenizer just fine.
There is 0 doubt that you are better of finetuning that model to use your tokenizer than training from scratch. So what Meta gives you for free massively helps you building your model, that's OSS to me.