News for hackers. Stuff that matters.
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De... by rgbrgb on Tuesday November 21, 2023 @09:42AM (#38364254)
I get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed.

[0]: https://www.freechat.run/

There are no comments on this comment yet.