Re:Llamafile is the new best way to run a LLM on your own co...
by iJohnDoe on Friday December 01, 2023 @10:09PM
(#38495525)
Is there any reason to run this if you already have LM Studio installed?
Which LLM to download for general questions/info that has good accuracy?
Separately, I couldn’t figure out how to use the NVIDIA GPU with LM Studio. Only wants to use the Intel card. I use NVIDIA with Diffusion with no problems if that makes any difference.
Something like a Mistral-7B Dolphin finetune actually is surprisingly useful, like GPT-3.5 in some respects. I imagine it would render ~10 tok/sec on an M2 for at least short bursts until throttling sets in.
As I mentioned in another post, try this out with LM Studio. Super simple GUI that even a non-tech person could probably figure out for finding/downloading/loading models w/ a ChatGPT like interface for chatting.
https://lmstudio.ai/