Ask HN: Recommendations for Local LLMs in 2024: Private and Offline?

Re:Ask HN: Recommendations for Local LLMs in 2024: Private a... by Havoc on Saturday April 06, 2024 @02:44PM (#39954469)

You need suitable hardware (ideally a 3090, 4090 or an Apple M device with a decent amount of mem).

Then set up software - ollama for easy mode (but less control) or text-generation-webui for more control.

After that you can just try models. The subreddit /r/localllama has whatever is flavour of the week. The Mixtral model at like Q3 quantization is probably a good starting point

[ Parent | on HN ]

Re:Ask HN: Recommendations for Local LLMs in 2024: Private a... by daghamm on Saturday April 06, 2024 @04:13PM (#39955098)

With ROCm 6 I think new-ish AMD GPUs give you much better bang for the buck.

https://ollama.com/blog/amd-preview

[ Parent | on HN ]

Re:Ask HN: Recommendations for Local LLMs in 2024: Private a... by Havoc on Sunday April 07, 2024 @06:48AM (#39959918)

Indeed...bought AMD shares on the bet that they'll catch up.

However this is still true only in a fairly narrow space - inference - once you stray off that very narrow path they're still a fair bit behind

[ Parent | on HN ]

Re:Ask HN: Recommendations for Local LLMs in 2024: Private a... by FezzikTheGiant on Monday April 08, 2024 @07:10PM (#39974643)

So I can run local LLMs on my apple M1 air?

[ Parent | on HN ]

Re:Ask HN: Recommendations for Local LLMs in 2024: Private a... by tharmas on Tuesday April 09, 2024 @02:17PM (#39982396)

Yes you can. Download LM studio and then search for a model such as Mistral-7B-Instruct-v0.2. LM studio will suggest variants of that model that suit your hardware to download. Here's a demonstration: https://www.youtube.com/watch?v=VXHryjPu52k

[ Parent | on HN ]