Many options for running Mistral models in your terminal using LLM

Re:Many options for running Mistral models in your terminal ... by williamstein on Monday December 18, 2023 @05:32PM (#38689338)

I tried Mixtral via ollama on my Apple M1 Max with 32GB of RAM, and was a total nonstarter. I ended up having to powercycle my machine. I then just used two L4 GPU's on Google Cloud (so 48GB of GPU RAM, see [1]) and it was very smooth and fast there.

[1] https://github.com/sagemathinc/cocalc-howto/blob/main/ollama...

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by jmorgan on Monday December 18, 2023 @06:59PM (#38690070)

Wow, as an author of the project I'm so sorry about you having to restart your computer. The memory management in Ollama needs a lot of improvement – will be working on this a bunch going forward. I also have a M1 32GB Mac and it's unfortunately just below the amount of memory Mixtral needs to run well (for now!)

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by ls612 on Monday December 18, 2023 @11:17PM (#38691749)

Will the Mixtral 8x7 run well on a 64GB M2 Max then?

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by simonw on Tuesday December 19, 2023 @12:16AM (#38692116)

I'm using an M2 64GB and Mixtral works pretty well.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by ls612 on Tuesday December 19, 2023 @10:41PM (#38705194)

It’s wild my laptop can run AI models better than my RTX 4090 desktop. Thanks for the info!

[ Parent | on HN ]