News for hackers. Stuff that matters.
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by ggerganov on Saturday March 11, 2023 @10:11AM (#35108860)
It currently uses only the CPU via ARM NEON intrinsics - no GPU, no ANE and no Apple Accelerate.

Plan is in the future to utilize respective SIMD intrinsics for other architectures (AVX, WASM SIMD, etc) and also add other more accurate quantization approaches. It's actually not a lot of work and I have most of the stuff ready, so hopefully soon!

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by lxe on Saturday March 11, 2023 @10:56AM (#35109365)
Are you observing higher tokens/second throughout on Apple silicon (you’ve been mentioning 20/second) than running it in PyTorch on a CUDA GPU such as a 3090?
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by ggerganov on Saturday March 11, 2023 @11:18AM (#35109636)
I haven't run the original PyTorch model not a single time!

I just look at the code and port it. I don't have the hardware to run it.

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by simonw on Saturday March 11, 2023 @11:32AM (#35109842)
Are you funded at all?

If you need extra hardware I'm sure the community could make that happen.

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by MacsHeadroom on Saturday March 11, 2023 @10:05PM (#35116022)
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments.

(20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x smaller than 65B)

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by lxe on Sunday March 12, 2023 @08:09PM (#35128606)
How are you getting 20 tokens/second? I'm getting 2.6 tokens/s on 3090 with int4 prequantized model. Is 4090 so much faster?