News for hackers. Stuff that matters.

A comment on Many options for running Mistral models in your terminal using LLM. Parent comment

Re:Many options for running Mistral models in your terminal ... by 3abiton on Tuesday December 19, 2023 @10:18AM (#38696620)
That's 10x speed increase. What's the secret behind apple M3? Faster clocked RAMs? Specific AI hardware?
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by bugglebeetle on Tuesday December 19, 2023 @12:44PM (#38698785)
Unified memory and optimizations in llama.cpp (which Ollama wraps).
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by ithkuil on Tuesday December 19, 2023 @02:53PM (#38700624)
Is that using the GPU?
[ Parent | on HN ]
Re:Many options for running Mistral models in your terminal ... by bugglebeetle on Tuesday December 19, 2023 @03:35PM (#38701137)
It can be variably configured. There are details in the repo, but llama.cpp makes use of Metal.
[ Parent | on HN ]