Re:Many options for running Mistral models in your terminal ...
by amrrs on Monday December 18, 2023 @04:14PM
(#38688401)
I ended up ordering a 36GB M3 for similar reason.
I currently run Mistral and a few mistral derivatives using Ollama with decent inference speed on a 2019 Intel Mac 32GB. So I assumed the new one with 32ish should do a better job.
I've tried vision model Llava as well, a bit more latency but works fine.
I currently run Mistral and a few mistral derivatives using Ollama with decent inference speed on a 2019 Intel Mac 32GB. So I assumed the new one with 32ish should do a better job.
I've tried vision model Llava as well, a bit more latency but works fine.
With Apple's own Mlx things might improve .