News for hackers. Stuff that matters.

A comment on Many options for running Mistral models in your terminal using LLM. Parent comment

Re:Many options for running Mistral models in your terminal ... by traverseda on Monday December 18, 2023 @08:58PM (#38690899)
Huh, I've fought with a few of these things on my laptop, no nvidia GPU, limited ram, etc.

This actually worked as advertised.

[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by 3abiton on Monday December 18, 2023 @09:42PM (#38691152)
What's the performance like (quality and speed wise)?
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by all2 on Monday December 18, 2023 @11:03PM (#38691675)
Mistral 7b is serviceable for short contexts. If you have a longer conversation, token generation can start to lag a lot.
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by tarruda on Tuesday December 19, 2023 @02:12AM (#38692766)
Yesterday I tried mixtral 7bx8 running on the CPU. With an Intel 11th gen chip and 64gb DDR4 at 3200mhz, I got around 2-4 tokens/second in a small context, this gets progressively slower as the context grows.

You would get a much better experience with apple silicon and lots of RAM

[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by yobanate on Tuesday December 19, 2023 @04:06AM (#38693420)
Can confirm. My M3 Max gets about 22t/s, putting the bottleneck BKAC.
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by 3abiton on Tuesday December 19, 2023 @10:18AM (#38696620)
That's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by bugglebeetle on Tuesday December 19, 2023 @12:44PM (#38698785)
Unified memory and optimizations in llama.cpp (which Ollama wraps).
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by ithkuil on Tuesday December 19, 2023 @02:53PM (#38700624)
Is that using the GPU?
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by bugglebeetle on Tuesday December 19, 2023 @03:35PM (#38701137)
It can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.
[ Parent | on HN ]