Many options for running Mistral models in your terminal using LLM

Re:Many options for running Mistral models in your terminal ... by M4v3R on Monday December 18, 2023 @03:29PM (#38687749)

Which honestly is the easiest option of them all if you own an Apple Silicon based Mac. You just download the ollama and then run `ollama run mixtral` (or choose a quantization from their models page if you don't have enough ram to run the defalt q4 model) and that's it.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by eurekin on Monday December 18, 2023 @03:48PM (#38688003)

I tried an hour ago and had a can't load model error. Everything up to date. Is there any special step?

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by mmcclure on Monday December 18, 2023 @03:53PM (#38688098)

Tried `ollama pull mixtral` just now and it seems to be working, albeit pretty slowly.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by Casteil on Monday December 18, 2023 @04:20PM (#38688467)

How much RAM do you have? Mixtral is a beast and the non quantized model wants 40GB+ of memory.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by eurekin on Monday December 18, 2023 @06:05PM (#38689652)

Ah, that might be it! I have only 32

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by ukuina on Tuesday December 19, 2023 @11:58PM (#38705605)

The q2 should fit.

[ Parent | on HN ]