News for hackers. Stuff that matters.

A comment on Many options for running Mistral models in your terminal using LLM. Parent comment

Re:Many options for running Mistral models in your terminal ... by Casteil on Monday December 18, 2023 @04:20PM (#38688467)
How much RAM do you have? Mixtral is a beast and the non quantized model wants 40GB+ of memory.
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by eurekin on Monday December 18, 2023 @06:05PM (#38689652)
Ah, that might be it! I have only 32
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by ukuina on Tuesday December 19, 2023 @11:58PM (#38705605)
The q2 should fit.
[ Parent | on HN ]