Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
by lxe on Saturday March 11, 2023 @10:56AM
(#35109365)
Are you observing higher tokens/second throughout on Apple silicon (you’ve been mentioning 20/second) than running it in PyTorch on a CUDA GPU such as a 3090?
[ Parent
| on HN ]
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
by ggerganov on Saturday March 11, 2023 @11:18AM
(#35109636)
I haven't run the original PyTorch model not a single time!
I just look at the code and port it. I don't have the hardware to run it.
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
by simonw on Saturday March 11, 2023 @11:32AM
(#35109842)
Are you funded at all?
If you need extra hardware I'm sure the community could make that happen.
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp
by MacsHeadroom on Saturday March 11, 2023 @10:05PM
(#35116022)
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments.
(20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x smaller than 65B)