Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by vanillax on Monday August 28, 2023 @12:41PM
(#37296862)
wodner if you can pair with https://github.com/getumbrel/llama-gpt
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by noiv on Monday August 28, 2023 @12:52PM
(#37297049)
Thx. Where can I send flowers to?
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by ignoramous on Tuesday August 29, 2023 @09:11AM
(#37307327)
To any person you're in a position to be kind to.
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by redox99 on Monday August 28, 2023 @02:04PM
(#37298099)
Is there a way to make it work with ooba+exllama? (much faster than llamacpp)
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by thelastparadise on Monday August 28, 2023 @03:15PM
(#37299089)
You should be able to turn on the API in booba:
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by redox99 on Monday August 28, 2023 @03:20PM
(#37299133)
But that API isn't OpenAI compatible AFAIK
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by k4rli on Monday August 28, 2023 @02:17PM
(#37298334)
Thanks, works nicely and easy to set up.
Is it possible to use GPU for this? With R9 7900x and 32GB RAM it takes 15-30sec to generate response. I have a 6900XT which might be more suited for this.
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by npsomaratna on Monday August 28, 2023 @02:37PM
(#37298619)
Yes. In the llama.cpp server command, specify the number of layers you'd like offloaded to your GPU via the -ngl parameter, e.g.:
./server -t 8 -m models/wizardcoder-python-34b-v1.0.Q4_K_S.gguf -c 16384 --mlock -ngl 60
(You might need to play around with the number of layers.)[Edit: make sure to compile llama.cpp with GPU support first, e.g., "make clean && LLAMA_CUBLAS=1 make -j"]
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by tayo42 on Monday August 28, 2023 @02:33PM
(#37298565)
One of the most annoying things about learning ai/ml for me right now is how much of this stuff is hidden behind people's comlanies and projects with to many emojis.
Like I can't find simple straight foward solutions or content that isn't tied back to a company.
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by jfoucher on Monday August 28, 2023 @04:43PM
(#37300171)
I'm a complete beginner regarding this stuff, so if I may ask, how would I go about downloading the relevant model (e.g. wizardcoder-python-34b-v1.0.Q4_K_S.gguf) I checked on Hugging face but all I got was a bunch of .bin files...
Thanks.
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by rpgwaiter on Monday August 28, 2023 @07:46PM
(#37301756)
Google the filename + "torrent download"
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by npsomaratna on Monday August 28, 2023 @09:45PM
(#37302543)
Do a search on the HuggingFace models page, e.g.:
https://huggingface.co/models?sort=trending&search=wizardcod...
The rough list of steps to do so are:
You can access your local coding LLM through the Continue sidebar now.