It’s in the near past. https://github.com/ggerganov/llama.cpp

Also worth checking out https://github.com/saharNooby/rwkv.cpp which is based on Georgi's library and offers support for the RWKV family of models which are Apache-2.0 licensed.

I’ve got some of their smaller Raven models running locally on my M1 (only 16GB of RAM).

I’m also in the middle of making it user friendly to run these models on all platforms (built with Flutter). First MacOS release will be out before this weekend: https://github.com/BrutalCoding/shady.ai

You can see for yourself (assuming you have the model weights) https://github.com/abetlen/llama-cpp-python

I get around ~140 ms per token running a 13B parameter model on a thinkpad laptop with an 14 core intel i7-9750 processor. Because it's CPU inference the initial prompt takes longer to process so total latency is still higher than I'd like but I'm working on some caching solutions that should make this bareable for things like chat.

I tried Vicuna on my laptop's CPU. I really wasn't impressed, it seemed terrible in comparison to the web version with a GPU and could only answer basic questions.

But I don't know how optimized all the parameters were, i was just using default everything from the nearest tutorial I could find.