News for hackers. Stuff that matters.
Re:The Coming of Local LLMs by abetlen on Tuesday April 11, 2023 @12:14PM (#35527261)
You can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python

I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with an 14 core intel i7-9750 processor. Because it's CPU inference the initial prompt takes longer to process so total latency is still higher than I'd like but I'm working on some caching solutions that should make this bareable for things like chat.

There are no comments on this comment yet.