Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by iAkashPaul on Tuesday November 21, 2023 @01:53AM
(#38360266)
llama.cpp can run the Q4 variant of the same at 30tok/s on an M1 Pro unlike the 20tok/s being quoted
[ Parent
| on HN ]
Re:Self-Hosting Open Source LLMs: Cross Devices and Local De...
by rgbrgb on Tuesday November 21, 2023 @09:42AM
(#38364254)
I get similar speeds on my m1, though it depends a bit on the quantization. If anyone else is on Mac and just want to have a play, FreeChat (my app) is a simple chat wrapper for llama.cpp. Clicking the little … on a message and checking the info will tell you the generation speed.