News for hackers. Stuff that matters.
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by npsomaratna on Monday August 28, 2023 @12:46PM (#37296955)
M1 with 32 GB RAM. I can just about fit the 4-bit quantized 33 GB Code Llama model (and it's finetunes, e.g. WizardCoder, etc.) into memory. It's somewhat slow, but good enough for my purposes.

Edit: when I bought my Macbook, I was like "Ok, I'll just take the base model and add another 16 GB of RAM. That should future proof it for at least another half-decade." Famous last words. Reminds me of the apocryphal story about how the designers of the 8088 CPU couldn't imagine why someone would ever need more than 1 MB of RAM...

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by filmgirlcw on Monday August 28, 2023 @01:28PM (#37297572)
This is why my rule for laptops with non-upgradable memory has been to max out the RAM at purchase -- and that has been my rule since 2012/2013 or whenever that trend really started.

(written from a 64GB M1 Pro Max)

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by joombaga on Monday August 28, 2023 @03:08PM (#37298993)
I wish they offered that much in the Air.
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by SparkyMcUnicorn on Monday August 28, 2023 @02:33PM (#37298564)
[delayed]
Re:Continue with LocalAI: An alternative to GitHub's Copilot... by npsomaratna on Monday August 28, 2023 @11:45PM (#37303343)
Thank you. I had to reduce the context length to get this to work without crashing (from 16k to 8k)β€”and I'm seeing the ~100% speed up you mentioned.

However, when I run the LLM, OSX becomes sluggish. I assume this is because the GPU's utilized to the point where hardware-based rendering slows down due to insufficient resources.

I wonder if there's a way to avoid that slowdown?

Re:Continue with LocalAI: An alternative to GitHub's Copilot... by SparkyMcUnicorn on Tuesday August 29, 2023 @12:15PM (#37310080)
I haven't noticed any slowdowns. Maybe check that threads/n_threads is set correctly for your machine (total cores - 2. 10 cores = 8, 8 cores = 6).

n_gpu_layers should also be set to anything other than 0 (default). I don't think the exact number matters for metal, but I use 128.