Re:Llama2.c: inference llama 2 in one file of pure C
by ljlolel on Monday July 24, 2023 @04:26AM
(#36844584)
1. Itβs faster and cheaper to train a smaller model
2. Better than tokens is to train on probability distributions (distillation) and trees of probability distributions
Re:Llama2.c: inference llama 2 in one file of pure C
by nickpsecurity on Friday July 28, 2023 @08:23AM
(#36905820)
I've never seen anything about training on probability distributions or trees of them. Do you have articles with examples you could share with us?
I did try a quick search for it. Found some interesting papers. The links to them are below in case anyone finds them interesting.
https://arxiv.org/abs/2212.11481
https://towardsdatascience.com/a-new-way-to-predict-probabil...
https://arxiv.org/pdf/1912.07913.pdf
https://dukespace.lib.duke.edu/dspace/bitstream/handle/10161...
That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you heavily restrict the range in which the model should operate.