News for hackers. Stuff that matters.
Re:Llama2.c: inference llama 2 in one file of pure C by GaggiX on Sunday July 23, 2023 @05:34PM (#36840086)
>My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.

That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you heavily restrict the range in which the model should operate.

Re:Llama2.c: inference llama 2 in one file of pure C by ljlolel on Monday July 24, 2023 @04:26AM (#36844584)
1. It’s faster and cheaper to train a smaller model

2. Better than tokens is to train on probability distributions (distillation) and trees of probability distributions

Re:Llama2.c: inference llama 2 in one file of pure C by nickpsecurity on Friday July 28, 2023 @08:23AM (#36905820)
I've never seen anything about training on probability distributions or trees of them. Do you have articles with examples you could share with us?

I did try a quick search for it. Found some interesting papers. The links to them are below in case anyone finds them interesting.

https://arxiv.org/abs/2212.11481

https://towardsdatascience.com/a-new-way-to-predict-probabil...

https://arxiv.org/pdf/1912.07913.pdf

https://dukespace.lib.duke.edu/dspace/bitstream/handle/10161...