Re:Llama2.c: inference llama 2 in one file of pure C
by akomtu on Sunday July 23, 2023 @05:50PM
(#36840214)
Random thought: right now an LLM returns a probabilities distribution, an RNG sampler picks one and apoends it to the output, then the sequence repeats; but can the RNG instead pick N tokens that approximate the distribution, ask LLM to generate N new distributions, combine them somehow, then pick another set of N tokens from the combined dustribution?
[ Parent
| on HN ]
Re:Llama2.c: inference llama 2 in one file of pure C
by quickthrower2 on Sunday July 23, 2023 @09:11PM
(#36841847)
Novice here.
I like the sound of that!
I don't know the answer but I might experiment with it. Probably a researcher has tried it.
You would need N times the compute per token generate of course.
You could either pick to N, or sample N (with temperature adjustment to logits if needed)
Re:Llama2.c: inference llama 2 in one file of pure C
by immibis on Monday July 24, 2023 @02:45AM
(#36843876)
Sounds like a good avenue to research, but you probably want to generate more than 2 tokens ahead. Try 20 tokens, but I suppose you don't want N^20 executions of the LLM, but more like a representative sampling of say 200 combinations of the next 20 tokens. I don't know how you'd do that.
Re:Llama2.c: inference llama 2 in one file of pure C
by jauntbox on Monday July 24, 2023 @04:14AM
(#36844506)
This sounds pretty much like beam search (https://en.wikipedia.org/wiki/Beam_search), which is in fact a common generation technique! See eg. https://huggingface.co/docs/transformers/internal/generation...