News for hackers. Stuff that matters.
Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by MacsHeadroom on Saturday March 11, 2023 @10:05PM (#35116022)
A 4090 gets 30 tokens/second with LLaMA-30B, which is about 10 times faster than the 300ms/token people are reporting in these comments.

(20 tokens/second on a Mac is for the smallest model, ~5x smaller than 30B and 10x smaller than 65B)

Re:Running LLaMA 7B on a 64GB M2 MacBook Pro with Llama.cpp by lxe on Sunday March 12, 2023 @08:09PM (#35128606)
How are you getting 20 tokens/second? I'm getting 2.6 tokens/s on 3090 with int4 prequantized model. Is 4090 so much faster?