News for hackers. Stuff that matters.
Re:Llamafile is the new best way to run a LLM on your own co... by youniverse on Friday December 01, 2023 @03:54PM (#38492228)
Anyone know if this (or any LLM) is worth running locally on a Macbook M2 Air?

I'll give it a shot over the weekend but if anyone knows I'm curious!

Re:Llamafile is the new best way to run a LLM on your own co... by brandall10 on Friday December 01, 2023 @04:21PM (#38492605)
If you have 8GB you can play around with heavily quantized 7B models, or up to moderately quantized 13GB models w/ 16GB.

Something like a Mistral-7B Dolphin finetune actually is surprisingly useful, like GPT-3.5 in some respects. I imagine it would render ~10 tok/sec on an M2 for at least short bursts until throttling sets in.

As I mentioned in another post, try this out with LM Studio. Super simple GUI that even a non-tech person could probably figure out for finding/downloading/loading models w/ a ChatGPT like interface for chatting.

https://lmstudio.ai/

Re:Llamafile is the new best way to run a LLM on your own co... by iJohnDoe on Friday December 01, 2023 @10:09PM (#38495525)
Is there any reason to run this if you already have LM Studio installed?

Which LLM to download for general questions/info that has good accuracy?

Separately, I couldn’t figure out how to use the NVIDIA GPU with LM Studio. Only wants to use the Intel card. I use NVIDIA with Diffusion with no problems if that makes any difference.

Re:Llamafile is the new best way to run a LLM on your own co... by buffington on Friday December 01, 2023 @04:13PM (#38492486)
Without knowing what you consider to be of worth, that's difficult to answer.

The good news is that the article describes in detail how to determine for yourself if it's worth it. The author uses an M2 as well, so you can at least know it'll likely work.

Re:Llamafile is the new best way to run a LLM on your own co... by MPSimmons on Friday December 01, 2023 @04:14PM (#38492497)
It depends on what you want them to do and what you want to get out of them. A lot of people use less machine than a M2 Air for different things.
Re:Llamafile is the new best way to run a LLM on your own co... by dataveg on Friday December 01, 2023 @06:16PM (#38493909)
I have Ollama running llama 7B on an 32GB Mac M1 Pro and its fast enough to be useful.