Re:Llama2.c: inference llama 2 in one file of pure C
by novaRom on Sunday July 23, 2023 @05:04PM
(#36839823)
I did use a tweaked nanoGPT to pretrain a 12M model on TinyStories (2Gbytes produced by GPT4), and results are pretty amazing. I've adapted it a bit on Wikipedia then, and it looks like a solid bullshit generator, much smarter than any smoothed n-gram model, and significantly smaller. My bet small LLMs will be predominant in multiple areas. My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
[ Parent
| on HN ]
Re:Llama2.c: inference llama 2 in one file of pure C
by GaggiX on Sunday July 23, 2023 @05:34PM
(#36840086)
>My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you heavily restrict the range in which the model should operate.
Re:Llama2.c: inference llama 2 in one file of pure C
by ljlolel on Monday July 24, 2023 @04:26AM
(#36844584)
1. It’s faster and cheaper to train a smaller model
2. Better than tokens is to train on probability distributions (distillation) and trees of probability distributions
Re:Llama2.c: inference llama 2 in one file of pure C
by nickpsecurity on Friday July 28, 2023 @08:23AM
(#36905820)
I've never seen anything about training on probability distributions or trees of them. Do you have articles with examples you could share with us?
I did try a quick search for it. Found some interesting papers. The links to them are below in case anyone finds them interesting.
https://arxiv.org/abs/2212.11481
https://towardsdatascience.com/a-new-way-to-predict-probabil...
https://arxiv.org/pdf/1912.07913.pdf
https://dukespace.lib.duke.edu/dspace/bitstream/handle/10161...
Re:Llama2.c: inference llama 2 in one file of pure C
by Remmy on Sunday July 23, 2023 @11:27PM
(#36842784)
Would love to read more about your time in NanoGPT. I've been getting familiar with it myself lately and it's still pretty much gibberish in the output with 16M, but the dataset is admittedly trash right now as well.
Re:Llama2.c: inference llama 2 in one file of pure C
by oaguy1 on Monday July 24, 2023 @04:49AM
(#36844748)
I also trained nanoGPT on TinyStories, produced about a 32M model. The results are amazing, especially considering I opted for a character-level model similar to the toy dataset in the repo. I’m writing about the experience while also doing a deep dive into the code on medium (username oaguy1). Smaller LLMs are definitely worth considering with the right quality training data. Once I finish playing with TinyStories, I recently tweaked the Standardized Project Gutenberg Corpus (~11GB) to be more modern. Want to see what I can do with it with nanoGPT and then maybe Huggingface’s libraries.