Ask HN: Recommendations for Local LLMs in 2024: Private and Offline?
I'm in search of a local LLM that can run completely offline for processing personal documents. Key requirements include privacy (no data leaves my machine) and performance (efficient with large datasets). Any recommendations for open-source / commercial solutions that fit the bill in 2024?
Also, what's the current state of local LLMs—are: Are they practical and useful, or still facing significant limitations?
(
29 comments
| on HN
)
They are. Working on a product using a fine-tuned Mistral-7B-Instruct-v0.2 model and it's pretty mind-blowing. Works flawlessly on my RTX3090 and serviceable on my M1 MBP as well. I'm building in Rust (using the candle crate), but for personal usage Python is probably the better choice since it's easier to get up and running.
I have infra in my house, not gonna lie it cost a lot, I have a rack with 30k of equipment in it (including 5 GPUs)
But this would probably run on an AWS P2 instance with is 0.90 USD an hour, or there’s lambda labs which is also pretty cheap (no affiliation, just satisfied customer)
The most expensive piece in either build is a video card. You want to be able to load the LLM file into your video card RAM.
I just got a 16gb new card for this. I can load up to 34b models on it but poorly. Anything 13b or less runs perfectly. A 12gb card would be able to run 7b models and with the right training I think 7b models can be awesome.
New 4060ti Used 3090
It works great, until recently WizardLM 2 came out and I tried it with local RAG. The difference is quite significant: https://twitter.com/chxy/status/1780101542311579942
Considering switching the default model after they put it back on HF.
https://ollama.ai
https://ollama.com
Then set up software - ollama for easy mode (but less control) or text-generation-webui for more control.
After that you can just try models. The subreddit /r/localllama has whatever is flavour of the week. The Mixtral model at like Q3 quantization is probably a good starting point
https://ollama.com/blog/amd-preview
However this is still true only in a fairly narrow space - inference - once you stray off that very narrow path they're still a fair bit behind
We've added support for it in our app if you wanna give it a try: https://curiosity.ai
RAG happens all locally (local embedding model and local vector db).
The app is secured by Mac App Sandbox, meaning it only have access to your selected file in the system dialog or drag and dropped files. If you use a local LLM, everything works offline.