I've had the best luck with WizardLM-7B. It runs at ~4 tokens/sec on my M1 and has decent outputs for simple situations. While it can't handle complex refactors like gpt4, I believe there's room to improve both via prompting and codebase-specific fine-tuning.

And good point! I've just added this as the recommendation in the README.

I tried to change config.py to use ggml setting, but I did not see any request going to local llama-cpp.server. It keeps on asking for openai key. The local llama-cpp.server is up and I was able to make swagger calls for completion.

Assuming that may be I missed something, I deleted ~/.continue and trying to start from scratch (vs code message: Starting Continue Server ...). I do see FastAPI is up on http://localhost:65432/ still vscode dialog shows same message. Where I can see the logs what its trying to do? What am I missing?

Anyone else able to use local ggml?

Hey there. I realized that the documentation is less clear than it should be on this, so I've made an update: https://continue.dev/docs/customization#local-models-with-gg...

The additional thing you need to do is set "default_model" to "ggml" in ~/.continue/config.py.