Re:Anthropic’s $5B, 4-year plan to take on OpenAI
by rnk on Tuesday April 11, 2023 @07:51PM
(#35533305)
It sounds like we are in a similar position. I had no desire to get a 64gb laptop from apple until all the interesting things from running llama locally came out. I wasn't even aware of the specific benefit of that uniform memory model on the mac. Now I'm looking at do I want to do 64, 96 or 128gb. For an insane amount of money, 5k for that top end one.
[ Parent
| on HN ]
Re:Anthropic’s $5B, 4-year plan to take on OpenAI
by anentropic on Wednesday April 12, 2023 @08:22AM
(#35538495)
The unified memory ought to be great for running LLaMA on the GPU on these Macbooks (since it can't run on the Neural Engine currently)
The point of llama.cpp is most people don't have a GPU with enough RAM, Apple unified memory ought to solve that
Some people have it working apparently:
Re:Anthropic’s $5B, 4-year plan to take on OpenAI
by rnk on Wednesday April 12, 2023 @01:12PM
(#35543131)
Thank you, that's exactly what I was looking for, specific info on perf.
Re:Anthropic’s $5B, 4-year plan to take on OpenAI
by anentropic on Wednesday April 12, 2023 @05:40PM
(#35547373)
I think the GPU performance for inference is probably limited currently by immaturity of PyTorch MPS (Metal) backend
before I found the repo above I had a naive attempt to get llama running with mps and it didn't "just work" - bunch of ops not supported etc