Many options for running Mistral models in your terminal using LLM

Re:Many options for running Mistral models in your terminal ... by robterrell on Monday December 18, 2023 @04:29PM (#38688580)

I've been unable to get Mixtral (mixtral-8x7b-instruct-v0.1.Q6_K.gguf) to run well on my M2 MacBook Air (24 GB). It's super slow and eventually freezes after about 12-15 tokens of a response. You should look at M3 options with more RAM -- 64 GB or even the weird-sounding 96 GB might be a good choice.

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by itissid on Monday December 18, 2023 @05:31PM (#38689326)

https://www.reddit.com/r/LocalLLaMA/comments/17kcgjv/how_doe... reddit thread talks about some of the pros and cons of a m3 max with 128 Gb costing ~5-6K

[ Parent | on HN ]

Re:Many options for running Mistral models in your terminal ... by mkesper on Tuesday December 19, 2023 @03:08AM (#38693088)

You can buy multiple 4090s for that money and will get real GPUs including tensor cores. Still relevant it seems: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

[ Parent | on HN ]