Many options for running Mistral models in your terminal using LLM

Re:Many options for running Mistral models in your terminal ... by azeirah on Monday December 18, 2023 @04:07PM (#38688283)

I was talking about m2 Macs. Just comparing that the best you can do with a gpu is 24GB, Macs go far beyond because of their integrated memory.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by MilaM on Monday December 18, 2023 @04:11PM (#38688341)

Ok, sorry. I did not understand that you just mentioned that to give more context. Totally makes sense.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by knotty66 on Tuesday December 19, 2023 @05:05AM (#38693808)

Just be aware you don't get to use all of it. I believe you only get access to ~20.8GB of GPU memory on a 32GB Apple Silicon Mac, and perhaps something like ~48GB on a 64GB Mac. I think there are some options to reconfigure the balance but they are technical and do not come without risk.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by MilaM on Tuesday December 19, 2023 @12:43PM (#38698757)

This is an important consideration. Thanks for mentioning it.

[ Parent | on HN ]