Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by rgbrgb on Tuesday November 21, 2023 @09:42AM
(#38364254)
I get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed.
[0]: https://www.freechat.run/