News for hackers. Stuff that matters.
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by lxe on Saturday March 11, 2023 @10:56AM (#35109365)
Are you observing higher tokens/second throughout on Apple silicon (you’ve been mentioning 20/second) than running it in PyTorch on a CUDA GPU such as a 3090?
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by ggerganov on Saturday March 11, 2023 @11:18AM (#35109636)
I haven't run the original PyTorch model not a single time!

I just look at the code and port it. I don't have the hardware to run it.

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by simonw on Saturday March 11, 2023 @11:32AM (#35109842)
Are you funded at all?

If you need extra hardware I'm sure the community could make that happen.

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by MacsHeadroom on Saturday March 11, 2023 @10:05PM (#35116022)
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments.

(20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x smaller than 65B)

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by lxe on Sunday March 12, 2023 @08:09PM (#35128606)
How are you getting 20 tokens/second? I'm getting 2.6 tokens/s on 3090 with int4 prequantized model. Is 4090 so much faster?