Re:Many options for running Mistral models in your terminal ...
by williamstein on Monday December 18, 2023 @05:32PM
(#38689338)
I tried Mixtral via ollama on my Apple M1 Max with 32GB of RAM, and was a total nonstarter. I ended up having to powercycle my machine. I then just used two L4 GPU's on Google Cloud (so 48GB of GPU RAM, see [1]) and it was very smooth and fast there.
[1] https://github.com/sagemathinc/cocalc-howto/blob/main/ollama...
Re:Many options for running Mistral models in your terminal ...
by jmorgan on Monday December 18, 2023 @06:59PM
(#38690070)
Wow, as an author of the project I'm so sorry about you having to restart your computer. The memory management in Ollama needs a lot of improvement – will be working on this a bunch going forward. I also have a M1 32GB Mac and it's unfortunately just below the amount of memory Mixtral needs to run well (for now!)
Re:Many options for running Mistral models in your terminal ...
by ls612 on Monday December 18, 2023 @11:17PM
(#38691749)
Will the Mixtral 8x7 run well on a 64GB M2 Max then?
In the article, Simon mentions the Q6_K.gguf model, which is about 40GB. A Mac Studio can handle this, but any of these models are going to be a tight fit or impossible on a Mac laptop without swapping to disk. Maybe NVME is fast enough that swapping isn't too terrible.
In my experience, the Mixtral models work pretty well on llama.cpp on my Linux workstation with a 10GB GPU, and offloading the rest to CPU.
It is impressive how fast the smaller models are improving. Still, a safe rule of thumb is the more RAM the better.
Also, really question how much you need to run these models locally. If you just want to play around with these models, it's probably far more cost effective to rent something in the cloud.