News for hackers. Stuff that matters.
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by willsmith72 on Monday August 28, 2023 @12:43PM (#37296908)
Can anyone share how the computer's performance is impacted by running the model locally? And what your specs are?
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by nico on Monday August 28, 2023 @12:48PM (#37296981)
Are there any LLMs that run on regular (AMD/Intel) CPUs? Or does everything require at least an M1 or a decent GPU?
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by loudmax on Monday August 28, 2023 @01:38PM (#37297715)
You can absolutely run LLMs without a GPU, but you need to set expectations for performance. Some projects to look into are

  * llama.cpp - https://github.com/ggerganov/llama.cpp
  * KoboldCpp - https://github.com/LostRuins/koboldcpp
  * GPT4All - https://gpt4all.io/index.html
llama.ccp will run LLMs that have been ported to the gguf format. If you have enough RAM, you can even run the big 70 billion parameter models. If you have a CUDA GPU, you can even offload part of the model onto the GPU and have the CPU do the rest, so you can get some partial performance benefit.

The issue is that the big models run too slowly on a CPU to feel interactive. Without a GPU, you'll get much more reasonable performance running a smaller 7 billion parameter model instead. The responses won't be as good as the larger models, but they may still be good enough to be worthwhile.

Also, development in this space is still coming extremely rapidly, especially for specialized models like ones tuned for coding.

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by sp332 on Monday August 28, 2023 @01:00PM (#37297172)
They do run, just slowly. Still better than nothing if you want to run something larger than would fit in your VRAM though. The Llama.ccp project is the most popular runtime, but I think all the major ones have a flag like "--cpu".
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by npsomaratna on Monday August 28, 2023 @12:46PM (#37296955)
M1 with 32 GB RAM. I can just about fit the 4-bit quantized 33 GB Code Llama model (and it's finetunes, e.g. WizardCoder, etc.) into memory. It's somewhat slow, but good enough for my purposes.

Edit: when I bought my Macbook, I was like "Ok, I'll just take the base model and add another 16 GB of RAM. That should future proof it for at least another half-decade." Famous last words. Reminds me of the apocryphal story about how the designers of the 8088 CPU couldn't imagine why someone would ever need more than 1 MB of RAM...

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by filmgirlcw on Monday August 28, 2023 @01:28PM (#37297572)
This is why my rule for laptops with non-upgradable memory has been to max out the RAM at purchase -- and that has been my rule since 2012/2013 or whenever that trend really started.

(written from a 64GB M1 Pro Max)

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by joombaga on Monday August 28, 2023 @03:08PM (#37298993)
I wish they offered that much in the Air.
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by SparkyMcUnicorn on Monday August 28, 2023 @02:33PM (#37298564)
[delayed]
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by npsomaratna on Monday August 28, 2023 @11:45PM (#37303343)
Thank you. I had to reduce the context length to get this to work without crashing (from 16k to 8k)—and I'm seeing the ~100% speed up you mentioned.

However, when I run the LLM, OSX becomes sluggish. I assume this is because the GPU's utilized to the point where hardware-based rendering slows down due to insufficient resources.

I wonder if there's a way to avoid that slowdown?

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by SparkyMcUnicorn on Tuesday August 29, 2023 @12:15PM (#37310080)
I haven't noticed any slowdowns. Maybe check that threads/n_threads is set correctly for your machine (total cores - 2. 10 cores = 8, 8 cores = 6).

n_gpu_layers should also be set to anything other than 0 (default). I don't think the exact number matters for metal, but I use 128.