News for hackers. Stuff that matters.
Re:Llamafile is the new best way to run a LLM on your own co... by brandall10 on Friday December 01, 2023 @04:21PM (#38492605)
If you have 8GB you can play around with heavily quantized 7B models, or up to moderately quantized 13GB models w/ 16GB.

Something like a Mistral-7B Dolphin finetune actually is surprisingly useful, like GPT-3.5 in some respects. I imagine it would render ~10 tok/sec on an M2 for at least short bursts until throttling sets in.

As I mentioned in another post, try this out with LM Studio. Super simple GUI that even a non-tech person could probably figure out for finding/downloading/loading models w/ a ChatGPT like interface for chatting.

https://lmstudio.ai/

Re:Llamafile is the new best way to run a LLM on your own co... by iJohnDoe on Friday December 01, 2023 @10:09PM (#38495525)
Is there any reason to run this if you already have LM Studio installed?

Which LLM to download for general questions/info that has good accuracy?

Separately, I couldn’t figure out how to use the NVIDIA GPU with LM Studio. Only wants to use the Intel card. I use NVIDIA with Diffusion with no problems if that makes any difference.