Re:Llama2.c: inference llama 2 in one file of pure C
by vikp on Sunday July 23, 2023 @04:55PM
(#36839739)
I would use textsynth (https://bellard.org/ts_server/) or llama.cpp (https://github.com/ggerganov/llama.cpp) if you're running on CPU.
Speed will be around 15 tokens per second on CPU (tolerable), and 5-10x faster with a GPU.
[ Parent
| on HN ]
There are no comments on this comment yet.