Performance of llama.cpp on Apple Silicon A-series

Re:Performance of llama.cpp on Apple Silicon A-series by yieldcrv on Tuesday December 19, 2023 @06:28PM (#38703422)

love that, I’ve been using Mistral 7B on my M1 and I thought it was tolerable but turned out I wasnt utilizing Metal and now its amazing

8x7B nowadays

As long as metal is used on an iphone I could see it worked well too. I use quantize 5 on my laptop but quantize 4 seems very practical

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by alberth on Tuesday December 19, 2023 @06:47PM (#38703599)

What did you do to enable Metal?

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by yieldcrv on Tuesday December 19, 2023 @06:58PM (#38703703)

I use LM Studio and 0.2.9 now has a checkmark for it

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by addandsubtract on Wednesday December 20, 2023 @08:17AM (#38708292)

How much RAM do you have? Ollama comes with a note: "this model requires 48GB of RAM".

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by moneywoes on Tuesday December 19, 2023 @07:50PM (#38704150)

could you please eli5 how metal works

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by yieldcrv on Tuesday December 19, 2023 @08:13PM (#38704319)

its just a co-processor included in Apple devices that has relevant components of a GPU, can accelerate graphics and general computation tasks

Apple’s chip architecture has a variety of co-processors alongside the CPU that make it the most perfect for transformers, in a convenient laptop form factor with fanless heat dissipation and low energy footprint

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by zakki on Tuesday December 19, 2023 @06:43PM (#38703567)

How do you use 8x7B?

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by localhost on Tuesday December 19, 2023 @07:11PM (#38703795)

Use ollama [1]

$ ollama run mixtral

[1] https://github.com/jmorganca/ollama

[ Parent | on HN ]

Re:Performance of llama.cpp on Apple Silicon A-series by yieldcrv on Tuesday December 19, 2023 @06:59PM (#38703711)

running it with LM Studio. No command line and has a huggingface browser, very user friendly.

[ Parent | on HN ]