* llama.cpp - https://github.com/ggerganov/llama.cpp
* KoboldCpp - https://github.com/LostRuins/koboldcpp
* GPT4All - https://gpt4all.io/index.html
llama.ccp will run LLMs that have been ported to the gguf format. If you have enough RAM, you can even run the big 70 billion parameter models. If you have a CUDA GPU, you can even offload part of the model onto the GPU and have the CPU do the rest, so you can get some partial performance benefit.The issue is that the big models run too slowly on a CPU to feel interactive. Without a GPU, you'll get much more reasonable performance running a smaller 7 billion parameter model instead. The responses won't be as good as the larger models, but they may still be good enough to be worthwhile.
Also, development in this space is still coming extremely rapidly, especially for specialized models like ones tuned for coding.
Edit: when I bought my Macbook, I was like "Ok, I'll just take the base model and add another 16 GB of RAM. That should future proof it for at least another half-decade." Famous last words. Reminds me of the apocryphal story about how the designers of the 8088 CPU couldn't imagine why someone would ever need more than 1 MB of RAM...
(written from a 64GB M1 Pro Max)
However, when I run the LLM, OSX becomes sluggish. I assume this is because the GPU's utilized to the point where hardware-based rendering slows down due to insufficient resources.
I wonder if there's a way to avoid that slowdown?
n_gpu_layers should also be set to anything other than 0 (default). I don't think the exact number matters for metal, but I use 128.