Many options for running Mistral models in your terminal using LLM

Re:Many options for running Mistral models in your terminal ... by MilaM on Monday December 18, 2023 @04:07PM (#38688286)

Thank you (and all the adjacent comments). I don't really need so much RAM for my regular work. Running the llms locally would only be for fun and experiments.

I think 32GB might be the best middle ground for my needs and budget constraints.

It's really a pity that you can't extend RAM in most Apple Silicon Macs and have to decide carefully upfront.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by amrrs on Monday December 18, 2023 @04:14PM (#38688401)

I ended up ordering a 36GB M3 for similar reason.

I currently run Mistral and a few mistral derivatives using Ollama with decent inference speed on a 2019 Intel Mac 32GB. So I assumed the new one with 32ish should do a better job.

I've tried vision model Llava as well, a bit more latency but works fine.

With Apple's own Mlx things might improve .

[ Parent | on HN ]