Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by filmgirlcw on Monday August 28, 2023 @01:28PM
(#37297572)
This is why my rule for laptops with non-upgradable memory has been to max out the RAM at purchase -- and that has been my rule since 2012/2013 or whenever that trend really started.
(written from a 64GB M1 Pro Max)
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by joombaga on Monday August 28, 2023 @03:08PM
(#37298993)
I wish they offered that much in the Air.
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by SparkyMcUnicorn on Monday August 28, 2023 @02:33PM
(#37298564)
[delayed]
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by npsomaratna on Monday August 28, 2023 @11:45PM
(#37303343)
Thank you. I had to reduce the context length to get this to work without crashing (from 16k to 8k)βand I'm seeing the ~100% speed up you mentioned.
However, when I run the LLM, OSX becomes sluggish. I assume this is because the GPU's utilized to the point where hardware-based rendering slows down due to insufficient resources.
I wonder if there's a way to avoid that slowdown?
Re:Continue with LocalAI: An alternative to GitHub's Copilot...
by SparkyMcUnicorn on Tuesday August 29, 2023 @12:15PM
(#37310080)
I haven't noticed any slowdowns. Maybe check that threads/n_threads is set correctly for your machine (total cores - 2. 10 cores = 8, 8 cores = 6).
n_gpu_layers should also be set to anything other than 0 (default). I don't think the exact number matters for metal, but I use 128.
Edit: when I bought my Macbook, I was like "Ok, I'll just take the base model and add another 16 GB of RAM. That should future proof it for at least another half-decade." Famous last words. Reminds me of the apocryphal story about how the designers of the 8088 CPU couldn't imagine why someone would ever need more than 1 MB of RAM...